Java CPU Flame Graphs

Update (September 14, 2015): Java 8 Update 60 was released on August 18, 2015. There are some recent changes to "perf-map-agent". There is no longer a script named "perf-java" and you should use "bin/create-java-perf-map.sh "

Brendan Gregg shared an exciting news in his Monitorama talk: The "JDK-8068945" is fixed in Java 8 Update 60 Build 19!

Without this fix, it was not possible to see full stack in Java with Linux perf_events and standard JDK (without any patches). For more information, see Brendan's Java CPU Flame Graphs page.

The Problem with Java and Perf

First of all, let's see what's the problem with using current latest Java and perf.

For this example, I used the same program explained in my previous blog post regarding FlameGraphs.

java org.wso2.example.JavaThreadCPUUsage.App

Then I sampled on-CPU functions for Java program using perf. (See Brendan's Perf Examples)

sudo perf record -F 99 -g -p `pgrep -f JavaThreadCPUUsage`

After few seconds, I pressed Ctrl+C to stop recording.

When I tried to list all raw events from "perf.data" using the "sudo perf script" command, I see following message.

Failed to open /tmp/perf-29463.map, continuing without symbols

Perf tries to load Java symbol table from the "/tmp/perf-29463.map" file. The 29463 part in the file name is the PID of the Java program.

Missing these Java symbols is one of the main problems with Java and Perf.  In here, there are actually two specific problems and Brendan has explained those in Java CPU Flame Graphs.

As explained by Brendan, in order to solve these problems, we need to provide Java symbol table for the perf and instruct JVM to preserve frame pointers. This is why we need the fix for  "JDK-8068945".

Generating Java CPU Flame Graphs

I downloaded the latest JDK™ 8u60 Early Access Release, which is "JDK 8u60 Build b24" as of now.

I extracted the JDK to a temp directory in my home. So, my JAVA_HOME is now "~/temp/jdk1.8.0_60".

With this release, I can use the JVM argument "-XX:+PreserveFramePointer" with java command.

~/temp/jdk1.8.0_60/bin/java -XX:+PreserveFramePointer org.wso2.example.JavaThreadCPUUsage.App

Now we need to create the Java symbol file. I found two ways to do that.

1. Use https://github.com/jrudolph/perf-map-agent
2. Use https://github.com/coderplay/perfj 

Using perf-map-agent

We need to build "perf-map-agent".

git clone https://github.com/jrudolph/perf-map-agent.git
cd perf-map-agent
export JAVA_HOME=~/temp/jdk1.8.0_60/
sudo apt-get install cmake
cmake .
make

Now we can start a perf recording. (Same command as mentioned above)

sudo perf record -F 99 -g -p `pgrep -f JavaThreadCPUUsage`

Now, run "bin/create-java-perf-map.sh".

bin/create-java-perf-map.sh `pgrep -f JavaThreadCPUUsage` 

This will create the Java symbol file in /tmp.

Please note that the perf-map-agent attaches to the java process. Please use the same JAVA_HOME to make sure there are no errors when attaching to the Java process.

Using perfj

Perfj is a wrapper of linux perf command for java programs. Download the latest release from perfj releases page.

tar -xvf perfj-1.0.tgz
cd perfj-1.0/
sudo -u isuru JAVA_HOME=/home/isuru/temp/jdk1.8.0_60 ./bin/perfj record -F 99 -g -p `pgrep -f JavaThreadCPUUsage`

Please note that I'm using the same user as the java process. This is required as perfj also attaches to the Java process. JAVA_HOME also must be same.

Generating Flame Graph

Now we have perf.data and java symbols to generate the flame graph.

sudo perf script | ~/performance/brendangregg-git/FlameGraph/stackcollapse-perf.pl > /tmp/out.perf-folded
cat /tmp/out.perf-folded | ~/performance/brendangregg-git/FlameGraph/flamegraph.pl --color=java --width 550 > /tmp/perf.svg

When you open the perf.svg in browser, you can see complete stack traces.

Summary

This is a quick blog post on Generating Java CPU Flame Graphs. To create Java symbol file, we can use perfj or perf-map-agent.

When I tested, perfj seems to provide better results.

Here is the flame graph generated with perfj java symbols!



If there are any questions, please ask in comments.