Friday, December 12, 2014

Java JIT Compilation, Inlining and JITWatch

Dr. Srinath recently shared an InfoQ article with us and its title is "Is Your Java Application Hostile to JIT Compilation?". I'm writing what I learnt from that article in this blog post.

Overview of Just-In-Time (JIT) compiler

Java code is usually compiled into platform independent bytecode (class files) using "javac" command. This "javac" command is the Java programming language compiler.

The JVM is able to load the class files and execute the Java bytecode via the Java interpreter. Even though this bytecode is usually interpreted, it might also be compiled in to native machine code using the JVM's Just-In-Time (JIT) compiler. 

Unlike the normal compiler, the JIT compiler compiles the code (bytecode) only when required. With JIT compiler, the JVM monitors the methods executed by the interpreter and identifies the “hot methods” for compilation. After identifying the Java method calls, the JVM compiles the bytecode into a more efficient native code.

In this way, the JVM can avoid interpreting a method each time during the execution and thereby improves the run time performance of the application.

The -client and -server systems

It's important to note that there are different JIT compilers for -client and -server systems. A server application needs to be run for a longer time and therefore it needs more optimizations. However a client application may not need lot of optimizations compared to a server application.


There are many optimization techniques for JIT compilation. Such optimization techniques are “inlining”, "dead code elimination" etc. 

Let's look at "inlining" optimization technique in this blog post.

The inlining process optimizes the code by substituting the body of a method into the places where that method is called.

Following are some advantages:

  • Eliminating the need for virtual method lookup
  • Save the cost of calling another method
  • Not needed to create a new stack frame
  • No performance penalty for good coding practices

Inlining depends on the method size. The value is configured by “-XX:MaxInlineSize” and the default value is 35 bytes.

For “hot” methods, which are called in high frequency, the threshold value is increased to 325 bytes. This threshold value is configured by “-XX:FreqInlineSize”.

JarScan tool in JITWatch

The JITWatch is an open source tool developed to get much better insight into how the JIT compiler affects the code.

JarScan is a tool included in JITWatch to analyze jar files and count the bytes of each method’s bytecode.

With this tool, we can identify the methods, which are too large to JIT.

PrintCompilation JVM Flag

The “-XX:+PrintCompilation” flag shows basic information about the HotSpot method compilation.

It generates logs like:
37    1      java.lang.String::hashCode (67 bytes)
124   2  s!  java.lang.ClassLoader::loadClass  (58 bytes)

In the example, the first columns show the time in milliseconds since the process started.

Second column is the compile ID, which track an individual method as it is compiled, optimized, and possibly deoptimized again.

The next column show additional information in the form of flags. (s - “synchronized”, ! - “has exception handlers”).

Last two columns show the the method name and the bytes of bytecode.

This flag doesn’t have much impact on JIT compiler performance and therefore we can use this flag in production.

We can use the PrintCompilation output and the JarScan output to determine which methods are compiled.

There are two minor problems with  PrintCompilation output.

  1. The method signature is not printed, which makes it difficult to identify overloaded methods.
  2. No way to configure log output to a different file.

Identifying JIT-friendly methods

Following is a simple process to determine whether methods are JIT-friendly.
  1. Identifying methods, which are in critical path for the transactions.
  2. JarScan output should not indicate such methods
  3. PrintCompilation output should show such methods being compiled.

Comparison of Java 7 and Java 8 methods

The InfoQ article compares the “$JAVA_HOME/jre/lib/rt.jar” of JDK 7 & 8 to identify the changes in inlining behaviour.

The Java 7 has 3684 inline-unfriendly methods and Java 8 has 3576 such methods. It’s important to know that methods like “split”, “toLowerCase”, &  “toUpperCase” in String are not inline-friendly in both Java versions. This is due to handling UTF-8 data rather than ASCII.


The JITWatch tool can analyze the compilation logs generated with the “-XX:+LogCompilation” flag.

The logs generated by LogCompilation are XML-based and has lot of information related to JIT compilation. Hence these files are very large.


This blog post is about the Just-In-Time (JIT) compiler and its "Inlining" optimization technique. The JIT compiler mainly helps to optimize run-time performance in HotSpot JVM

With JITWatch tools and PrintCompilation, we can understand the JIT behaviour in our applications. With a quick analysis we can figure out performance impacts.

The important point is that if a method is too large, the inlining optimization will not be used. Therefore it's important to write JIT-friendly methods when we consider the performance of a system.

It’s also important to measure the performance of original system and compare after applying fixes. We should never apply any performance driven changes blindly.

Tuesday, December 9, 2014

Oracle Java Installation script for Ubuntu

Few months ago, I wrote a blog post on Installing Oracle JDK 7 (Java Development Kit) on Ubuntu. It has several steps to install the JDK on Ubuntu.

Every time when there is a new version, I upgrade the Java version in my laptop. Since I do few repetitive steps for every Java installation, I wrote a simple installation script for Java.

The installation script is available at GitHub:

You just have to run "" with root privileges once you download the JDK from Oracle. It also supports the installation of JDK Demos and "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files"

Please refer the README at GitHub. 

The script supports JDK 7 and JDK 8. Please try the installation scripts and let me know any feedback! :)

Enabling Java Security Manager for WSO2 products

Why Java Security Manager is needed?

In Java, the Security Manager is available for applications to have various security policies. The Security Manager helps to prevent untrusted code from doing malicious actions on the system. 

You need to enable Security Manager, if you plan to host any untrusted user applications in WSO2 products, especially in products like WSO2 Application Server.

The security policies should explicitly allow actions performed by the code base. If any of the actions are not allowed by the security policy, there will be a SecurityException

For more information on this, you can refer Java SE 7 Security Documentation.

Security Policy Guidelines for WSO2 Products

When enabling Security Manager for WSO2 products, it is recommended to give all permissions to all jars inside WSO2 product. For that, we plan to sign all jars using a common key and grant all permissions to the signed code by using "signedBy" grant as follows.

grant signedBy "<signer>" {

We also recommend to allow all property reads and WSO2 has a customized Carbon Security Manager to deny certain system properties.

One of the main reasons is that in Java Security Policy, we need to explicitly mention which properties are allowed and if there are various user applications, we cannot have a pre-defined list of System Properties. Therefore Carbon Security Manager's approach is to define a list of denied properties using the System Property "". This approach basically changes Java Security Manager's rule of "Deny all, allow specified" to "Allow all, deny specified".

There is another system property named "restricted.packages" to control the package access. However this "restricted.packages" system property is not working in latest Carbon and I have created CARBON-14967 JIRA to fix that properly in a future Carbon release.

Signing all JARs inside WSO2 product.

To sign the jars, we need a key. We can use the keytool command to generate a key.

$ keytool -genkey -alias signFiles -keyalg RSA -keystore signkeystore.jks -validity 3650 -dname "CN=Isuru,OU=Engineering, O=WSO2, L=Colombo, ST=Western, C=LK"
Enter keystore password:  
Re-enter new password: 
Enter key password for
 (RETURN if same as keystore password):

Above keytool command creates a new keystore file. If you omit -dname argument, all key details will be prompted.

Now extract the WSO2 product. I will be taking WSO2 Application Server as an example.

$ unzip -q  ~/wso2-packs/

Let's create two scripts to sign the jars. First script will find all jars and the second script will be used to sign a jar using the keystore we created earlier. script:

if [[ ! -d $1 ]]; then
    echo "Please specify a target directory"
    exit 1
for jarfile in `find . -type f -iname \*.jar`
   ./ $jarfile
done script:


set -e



signjar="$JAVA_HOME/bin/jarsigner -sigalg MD5withRSA -digestalg SHA1 -keystore $keystore_file -storepass $keystore_storepass -keypass $keystore_keypass"
verifyjar="$JAVA_HOME/bin/jarsigner -keystore $keystore_file -verify"

echo "Signing $jarfile"
$signjar $jarfile $keystore_keyalias

echo "Verifying $jarfile"
$verifyjar $jarfile

# Check whether the verification is successful.
if [ $? -eq 1 ]
    echo "Verification failed for $jarfile"

Now we can see following files.

$ ls -l
-rwxrwxr-x  1 isuru isuru    602 Dec  9 13:05
-rwxrwxr-x  1 isuru isuru    174 Dec  9 12:56
-rw-rw-r--  1 isuru isuru   2235 Dec  9 12:58 signkeystore.jks
drwxr-xr-x 11 isuru isuru   4096 Dec  6  2013 wso2as-5.2.1

When we run, all JARs found inside WSO2 Application Server will be signed using the "signFiles" key.

$ ./ wso2as-5.2.1/ > log

Configuring WSO2 Product to use Java Security Manager

To configure Java Security Manager, we need to pass few arguments to the main Java process. 

Java Security Manager can be enabled by using "" system property. We will specify the WSO2 Carbon Security Manager using this argument.

We also need to specify the security policy file using "" system property.

As I mentioned earlier, we will also set "restricted.packages" & "" system properties.

Following is the recommended set of values to be used in (Edit the startup script and add following lines just before the line " org.wso2.carbon.bootstrap.Bootstrap $*" \$CARBON_HOME/repository/conf/sec.policy \
    -Drestricted.packages=sun.,,com.sun.xml.internal.bind.,com.sun.imageio.,org.wso2.carbon. \,, \

Exporting signFiles public key certificate and importing it to wso2carbon.jks

We need to import the signFiles public key certificate to the wso2carbon.jks as the security policy file will be referring the signFiles signer certificate from the wso2carbon.jks (as specified by the first line).

$ keytool -export -keystore signkeystore.jks -alias signFiles -file sign-cert.cer
$ keytool -import -alias signFiles -file sign-cert.cer -keystore wso2as-5.2.1/repository/resources/security/wso2carbon.jks

Note: wso2carbon.jks' keystore password is "wso2carbon".

The Security Policy File

As specified in the system property "", we will keep the security policy file at $CARBON_HOME/repository/conf/sec.policy

Following policy file should be enough for starting up WSO2 Application Server and deploying a sample JSF & CXF webapps.

keystore "file:${user.dir}/repository/resources/security/wso2carbon.jks", "JKS";

// ========= Carbon Server Permissions ===================================
grant {
    // Allow socket connections for any host
    permission "*:1-65535", "connect,resolve";

    // Allow to read all properties. Use in to restrict properties
    permission java.util.PropertyPermission "*", "read";
    permission java.lang.RuntimePermission "getClassLoader";
    // CarbonContext APIs require this permission
    permission "control";

    // Required by any component reading XMLs. For example: org.wso2.carbon.databridge.agent.thrift:4.2.1.
    permission java.lang.RuntimePermission "";

    // Required by org.wso2.carbon.ndatasource.core:4.2.0. This is only necessary after adding above permission. 
    permission java.lang.RuntimePermission "";

// ========= Platform signed code permissions ===========================
grant signedBy "signFiles" {

// ========= Granting permissions to webapps ============================
grant codeBase "file:${carbon.home}/repository/deployment/server/webapps/-" {

    // Required by webapps. For example JSF apps.
    permission java.lang.reflect.ReflectPermission "suppressAccessChecks";

    // Required by webapps. For example JSF apps require this to initialize com.sun.faces.config.ConfigureListener
    permission java.lang.RuntimePermission "setContextClassLoader";

    // Required by webapps to make HttpsURLConnection etc.
    permission java.lang.RuntimePermission "modifyThreadGroup";

    // Required by webapps. For example JSF apps need to invoke annotated methods like @PreDestroy
    permission java.lang.RuntimePermission "accessDeclaredMembers";

    // Required by webapps. For example JSF apps
    permission java.lang.RuntimePermission "";

    // Required by webapps. For example JSF EL
    permission java.lang.RuntimePermission "getClassLoader";

    // Required by CXF app. Needed when invoking services
    permission javax.xml.bind.JAXBPermission "setDatatypeConverter";

    // File reads required by JSF (Sun Mojarra & MyFaces require these)
    // MyFaces has a fix   
    permission "/META-INF", "read";
    permission "/META-INF/-", "read";

    // OSGi permissions are requied to resolve bundles. Required by JSF
    permission org.osgi.framework.AdminPermission "*", "resolve,resource";

The security policies may vary depending on your requirements. I recommend to test your application thoroughly in a development environment.

NOTE: There are risks in allowing some Runtime Permissions. Please look at the java docs for RuntimePermission. See Concerns below.

Troubleshooting Java Security

Java provides the "" system property to set various debugging options and monitor security access.

I recommend to add following line to whenever you need to troubleshoot some issue with Java Security."access,failure"

After adding that line, all the debug information will be printed to standard output. To check the logs, we can start the server using nohup.

$ nohup ./ &

Then we can grep the nohup.out and look for access denied messages.

$ tailf nohup.out | grep denied

Concerns with Java Security Policy

There are few concerns with current permission model in WSO2 products.

  • Use of ManagementPermission instead of Carbon specific permissions. The real ManagementPermission is used for a different purpose. I created CARBON-14966 jira to fix that.
  • Ideally the permission ' "control"' should not be specified in policy file as it is only required for privileged actions in Carbon. However due to indirect usage of such privileged actions within Carbon code, we need to specify that permission. This also needs to be fixed.
  • In above policy file, the JSF webapps etc require some risky runtime permissions. I recommend to use a Custom Runtime Environment (CRE) in WSO2 Application Server for JSF webapps etc and sign the jars inside CRE. You can also grant permissions based on the jar names (Use grant codeBase). However signing jars and using a CRE is a better approach with WSO2 AS.
If you also encounter any issues when using Java Security Manager, please discuss those issues in our developer mailing list.