Linux Performance Observability Tools

I am learning about Linux Performance Tools and I found Brendan Gregg's talks on Linux Performance are very interesting.

There are so many performance tools for Linux. Brendan recommends to follow a performance analysis methodology to analyze system or application performance. These methodologies can guide us to choose and use these performance tools effectively.

Linux Performance Observability Tools



There are different types of command line tools available in Linux. In this blog post, I'm going to focus on Linux Performance Observability Tools. I highly recommend to watch Brendan's talk at Velocity 2015 on Linux Performance Tools and I took details about following tools from his presentation and his website.


http://www.brendangregg.com/Perf/linux_observability_tools.png
Linux Performance Observability Tools
Taken from Brendan Gregg's Website: http://www.brendangregg.com/Perf/linux_observability_tools.png

Here are some examples of using Linux Performance Observability Tools in Ubuntu.  I tested each of these commands in a Ubuntu Trusty Vagrant Box

Basic Observability Tools



# Print load averages
uptime


# System and per-process interval summary. 
# It's important to note that the top can miss short-lived processes. 
# See 30 Linux TOP Command Examples With Screenshots
top


# htop is an interactive process viewer and you need to install it.
sudo apt-get install htop
htop



# Process status listing
ps -ef
# ASCII art forest
ps -ef f



# Virtual Memory Statistics
vmstat
# Show stats in Megabytes and update every second
vmstat -SM 1



# Block I/O (disk) stats. The iostat tool is in 'sysstat' package.
sudo apt-get install sysstat
iostat
# Display extended statistics in megabytes per seconds. 
# This also shows device utilization and omits the inactive devices
# during the sample period.
iostat -xmdz 1



# Report multi-processor statistics.
mpstat
# per-CPU stats
mpstat -P ALL 1



# Main memory usage in megabytes.
free -m



Intermediate Observability Tools



# Trace system calls and signals. This is not recommended in production.
strace
# Trace system calls in a process. 
# Prints the time (us) since epoch (-ttt) and syscall time (-T).
# Need to use sudo to attach to the process.
sudo strace -tttT -p 3344



# Sniff network packets for post analysis. 
# Using sudo to get permissions to capture packets on device
sudo tcpdump -i eth0 -w /tmp/out.tcpdump
# Read the dump
sudo tcpdump -nr /tmp/out.tcpdump



# Print network connections. See 10 basic examples of linux netstat command
netstat
# Network statistics by protocol
netstat -s
# Show both listening and not-listening sockets
netstat -a
# Show listening sockets
netstat -l
# Show the PID (-p) and the name of the program for TCP sockets (-t)
netstat -tp
# Kernel IP routing table
netstat -r
# Kernel Interface table
netstat -i



# Print network traffic statistics. You need to install 'nicstat' package.
sudo apt-get install nicstat
nicstat



# Process Stats
pidstat
# Process Stats by thread
pidstat -t
# Process Stats by disk I/O
pidstat -d



# Show swap usage
swapon -s
# Show swap usage in verbose mode
swapon -v



# List open file. Can be used as a debug tool
lsof
# Show active network connections
lsof -iTCP -sTCP:ESTABLISHED



# System Activity Reporter. 
# Before using sar, we need to enable data collection.
# See:
# Simple steps to install and configure sysstat/sar on Ubuntu/Debian server
# 10 Useful Sar (Sysstat) Examples for UNIX / Linux Performance Monitoring
sar -q



http://www.brendangregg.com/Perf/linux_observability_sar.png
Linux Performance Observability: sar
Taken from Brendan Gregg's Website: http://www.brendangregg.com/Perf/linux_observability_sar.png





Advanced Observability Tools



# Socket Statistics. This is similar to netstat,
# but it can display more TCP and state informations than other tools.
ss
# Socket Statistics. Show timer, processes and memory
ss -mop
# Show internal TCP information
ss -i



# Interactive Colorful IP LAN Monitor
sudo apt-get install iptraf
sudo iptraf



# Monitor I/O. A top-like tool
sudo apt-get install iotop
sudo iotop



# Kernel slab allocator memory usage
sudo slabtop



# Page cache statistics. 
# This tools is available in GitHub: https://github.com/tobert/pcstat
# and we need to use Go to build it.
# You can also download a binary from GitHub. Refer README for more information.
./pcstat testfile



# perf_events: Linux profiling with performance counters.
# This tool needs to be installed.
# I used perf command in previous blog post about Java CPU Flame Graphs. 
# See Brendan's Linux Perf Examples.
# Perf Tutorial is also good resource to learn about perf.
sudo apt-get install linux-tools-common linux-tools-generic
# List perf event
sudo perf list


# tiptop: reads hardware performance counters
# and displays statistics about running processes, such as IPC, or cache misses. 
# This tool was not available in Ubuntu 14.04 package repositories.
# Therefore I tried it on Ubuntu 15.04
sudo apt-get install tiptop
tiptop



# The rdmsr command reads a Model-Specific Register
# (MSR) value from the specified address.
# This tools is available from "msr-tools" package.
sudo apt-get install msr-tools
# Brendan has developed some Model Specific Register (MSR) 
# tools for Xen guests (eg, AWS EC2).
# Reading CPU temperature:
sudo rdmsr -p1 -f 23:16 -d 0x1a2




Summary


This blog post lists some Linux Performance Observability Tools. I have also linked man pages and some examples of using the tools.

As I mentioned, Brendan's presentations have more details on these tools and I just wanted to list those in one page for my own reference.
Location: Colombo, Sri Lanka

Comments

Post a Comment

Popular posts from this blog

Specifying a custom Event Settings file for Java Flight Recorder

Image
When you are using Java Flight Recorder (JFR), the JFR will use an event settings file to check which event types to record. By default in JFR, there are two settings, "default" and "profile". The default setting is recommended for Continuous Recordings as it has very low overhead (typically less than 1% overhead). The profile setting has more events and useful when profiling the application. As mentioned in my previous blog post regarding  Java Flight Recorder Continuous Recordings , we use following arguments to do a Continuous Recording. -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:FlightRecorderOptions=defaultrecording=true,disk=true,repository=./tmp,dumponexit=true,dumponexitpath=./ Note: According to the Oracle documentation on " java " command, we should be able to specify "settings" parameter to the -XX:FlightRecorderOptions. However, the settings parameter has no effect when used with the -XX:FlightRecorderOptions and ...
Read more

Flame Graphs with Java Flight Recordings

Flame Graphs Brendon D. Gregg , who is a computer performance analyst , has created  Flame Graphs to visualize stack traces in an interactive way. You must watch his talk at USENIX/LISA13 , titled Blazing Performance with Flame Graphs , which explains Flame Graphs in detail. There can be different types of flame graphs and I'm focusing on  CPU Flame Graphs  with Java in this blog post. Please look at the Flame Graphs Description  to understand the Flame Graph visualization. CPU Flame Graphs and Java Stack Traces As  Brendon  mentioned in his talk, understanding why CPUs are busy is very important when analyzing performance.  CPU Flame Graphs  is a good way to identify hot methods from sampled stack traces. In order to generate CPU Flame Graphs for Java Stack Traces , we need a way to get sample stack traces. Brendon has given examples to use jstack  and Google's lightweight-java-profiler . Please r...
Read more

Java Flight Recorder Continuous Recordings

When we are trying to find performance issues, it is sometimes necessary to do continuous recordings with Java Flight Recorder. Usually we debug issues in an environment similar to a production setup. That means we don't have a desktop environment and we cannot use Java Mission Control for flight recording. That also means we need to record & get dumps using command line in servers. We can of course use remote connection methods, but it's more easier to get recordings from the server. With continuous recordings, we need to figure out how to get dumps. There are few options. Get a dump when the Java application exits. For this, we need to use dumponexit and dumponexitpath options. Get a dump manually from JFR.dump diagnostic command via " jcmd " Note: The "jcmd" command is in $JAVA_HOME/bin. If you use the  Oracle Java Installation script for Ubuntu , you can directly use "jcmd" without including  $JAVA_HOME/bin in $PATH. Enabl...
Read more