Friday, December 12, 2014

Java JIT Compilation, Inlining and JITWatch

Dr. Srinath recently shared an InfoQ article with us and its title is "Is Your Java Application Hostile to JIT Compilation?". I'm writing what I learnt from that article in this blog post.

Overview of Just-In-Time (JIT) compiler


Java code is usually compiled into platform independent bytecode (class files) using "javac" command. This "javac" command is the Java programming language compiler.

The JVM is able to load the class files and execute the Java bytecode via the Java interpreter. Even though this bytecode is usually interpreted, it might also be compiled in to native machine code using the JVM's Just-In-Time (JIT) compiler. 

Unlike the normal compiler, the JIT compiler compiles the code (bytecode) only when required. With JIT compiler, the JVM monitors the methods executed by the interpreter and identifies the “hot methods” for compilation. After identifying the Java method calls, the JVM compiles the bytecode into a more efficient native code.

In this way, the JVM can avoid interpreting a method each time during the execution and thereby improves the run time performance of the application.

The -client and -server systems


It's important to note that there are different JIT compilers for -client and -server systems. A server application needs to be run for a longer time and therefore it needs more optimizations. However a client application may not need lot of optimizations compared to a server application.


Inlining


There are many optimization techniques for JIT compilation. Such optimization techniques are “inlining”, "dead code elimination" etc. 

Let's look at "inlining" optimization technique in this blog post.

The inlining process optimizes the code by substituting the body of a method into the places where that method is called.

Following are some advantages:

  • Eliminating the need for virtual method lookup
  • Save the cost of calling another method
  • Not needed to create a new stack frame
  • No performance penalty for good coding practices


Inlining depends on the method size. The value is configured by “-XX:MaxInlineSize” and the default value is 35 bytes.

For “hot” methods, which are called in high frequency, the threshold value is increased to 325 bytes. This threshold value is configured by “-XX:FreqInlineSize”.

JarScan tool in JITWatch


The JITWatch is an open source tool developed to get much better insight into how the JIT compiler affects the code.

JarScan is a tool included in JITWatch to analyze jar files and count the bytes of each method’s bytecode.

With this tool, we can identify the methods, which are too large to JIT.

PrintCompilation JVM Flag


The “-XX:+PrintCompilation” flag shows basic information about the HotSpot method compilation.

It generates logs like:
37    1      java.lang.String::hashCode (67 bytes)
124   2  s!  java.lang.ClassLoader::loadClass  (58 bytes)



In the example, the first columns show the time in milliseconds since the process started.

Second column is the compile ID, which track an individual method as it is compiled, optimized, and possibly deoptimized again.

The next column show additional information in the form of flags. (s - “synchronized”, ! - “has exception handlers”).

Last two columns show the the method name and the bytes of bytecode.

This flag doesn’t have much impact on JIT compiler performance and therefore we can use this flag in production.

We can use the PrintCompilation output and the JarScan output to determine which methods are compiled.

There are two minor problems with  PrintCompilation output.

  1. The method signature is not printed, which makes it difficult to identify overloaded methods.
  2. No way to configure log output to a different file.


Identifying JIT-friendly methods


Following is a simple process to determine whether methods are JIT-friendly.
  1. Identifying methods, which are in critical path for the transactions.
  2. JarScan output should not indicate such methods
  3. PrintCompilation output should show such methods being compiled.


Comparison of Java 7 and Java 8 methods


The InfoQ article compares the “$JAVA_HOME/jre/lib/rt.jar” of JDK 7 & 8 to identify the changes in inlining behaviour.

The Java 7 has 3684 inline-unfriendly methods and Java 8 has 3576 such methods. It’s important to know that methods like “split”, “toLowerCase”, &  “toUpperCase” in String are not inline-friendly in both Java versions. This is due to handling UTF-8 data rather than ASCII.

JITWatch


The JITWatch tool can analyze the compilation logs generated with the “-XX:+LogCompilation” flag.

The logs generated by LogCompilation are XML-based and has lot of information related to JIT compilation. Hence these files are very large.

Summary


This blog post is about the Just-In-Time (JIT) compiler and its "Inlining" optimization technique. The JIT compiler mainly helps to optimize run-time performance in HotSpot JVM

With JITWatch tools and PrintCompilation, we can understand the JIT behaviour in our applications. With a quick analysis we can figure out performance impacts.

The important point is that if a method is too large, the inlining optimization will not be used. Therefore it's important to write JIT-friendly methods when we consider the performance of a system.

It’s also important to measure the performance of original system and compare after applying fixes. We should never apply any performance driven changes blindly.

Tuesday, December 9, 2014

Oracle Java Installation script for Ubuntu

Few months ago, I wrote a blog post on Installing Oracle JDK 7 (Java Development Kit) on Ubuntu. It has several steps to install the JDK on Ubuntu.

Every time when there is a new version, I upgrade the Java version in my laptop. Since I do few repetitive steps for every Java installation, I wrote a simple installation script for Java.

The installation script is available at GitHub: https://github.com/chrishantha/install-java

You just have to run "install-java.sh" with root privileges once you download the JDK from Oracle. It also supports the installation of JDK Demos and "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files"

Please refer the install-java.sh README at GitHub. 

The script supports JDK 7 and JDK 8. Please try the installation scripts and let me know any feedback! :)



Enabling Java Security Manager for WSO2 products

Why Java Security Manager is needed?


In Java, the Security Manager is available for applications to have various security policies. The Security Manager helps to prevent untrusted code from doing malicious actions on the system. 

You need to enable Security Manager, if you plan to host any untrusted user applications in WSO2 products, especially in products like WSO2 Application Server.

The security policies should explicitly allow actions performed by the code base. If any of the actions are not allowed by the security policy, there will be a SecurityException

For more information on this, you can refer Java SE 7 Security Documentation.

Security Policy Guidelines for WSO2 Products


When enabling Security Manager for WSO2 products, it is recommended to give all permissions to all jars inside WSO2 product. For that, we plan to sign all jars using a common key and grant all permissions to the signed code by using "signedBy" grant as follows.

grant signedBy "<signer>" {
  permission java.security.AllPermission;
};

We also recommend to allow all property reads and WSO2 has a customized Carbon Security Manager to deny certain system properties.

One of the main reasons is that in Java Security Policy, we need to explicitly mention which properties are allowed and if there are various user applications, we cannot have a pre-defined list of System Properties. Therefore Carbon Security Manager's approach is to define a list of denied properties using the System Property "denied.system.properties". This approach basically changes Java Security Manager's rule of "Deny all, allow specified" to "Allow all, deny specified".

There is another system property named "restricted.packages" to control the package access. However this "restricted.packages" system property is not working in latest Carbon and I have created CARBON-14967 JIRA to fix that properly in a future Carbon release.

Signing all JARs inside WSO2 product.


To sign the jars, we need a key. We can use the keytool command to generate a key.

$ keytool -genkey -alias signFiles -keyalg RSA -keystore signkeystore.jks -validity 3650 -dname "CN=Isuru,OU=Engineering, O=WSO2, L=Colombo, ST=Western, C=LK"
Enter keystore password:  
Re-enter new password: 
Enter key password for
 (RETURN if same as keystore password):

Above keytool command creates a new keystore file. If you omit -dname argument, all key details will be prompted.

Now extract the WSO2 product. I will be taking WSO2 Application Server as an example.

$ unzip -q  ~/wso2-packs/wso2as-5.2.1.zip


Let's create two scripts to sign the jars. First script will find all jars and the second script will be used to sign a jar using the keystore we created earlier.

signJars.sh script:

#!/bin/bash
if [[ ! -d $1 ]]; then
    echo "Please specify a target directory"
    exit 1
fi
for jarfile in `find . -type f -iname \*.jar`
do
   ./signJar.sh $jarfile
done 


signJar.sh script:

#!/bin/bash

set -e

jarfile=$1

keystore_file="signkeystore.jks"
keystore_keyalias='signFiles'
keystore_storepass='wso2123'
keystore_keypass='wso2123'

signjar="$JAVA_HOME/bin/jarsigner -sigalg MD5withRSA -digestalg SHA1 -keystore $keystore_file -storepass $keystore_storepass -keypass $keystore_keypass"
verifyjar="$JAVA_HOME/bin/jarsigner -keystore $keystore_file -verify"

echo "Signing $jarfile"
$signjar $jarfile $keystore_keyalias

echo "Verifying $jarfile"
$verifyjar $jarfile

# Check whether the verification is successful.
if [ $? -eq 1 ]
then
    echo "Verification failed for $jarfile"
fi


Now we can see following files.

$ ls -l
-rwxrwxr-x  1 isuru isuru    602 Dec  9 13:05 signJar.sh
-rwxrwxr-x  1 isuru isuru    174 Dec  9 12:56 signJars.sh
-rw-rw-r--  1 isuru isuru   2235 Dec  9 12:58 signkeystore.jks
drwxr-xr-x 11 isuru isuru   4096 Dec  6  2013 wso2as-5.2.1

When we run signJars.sh, all JARs found inside WSO2 Application Server will be signed using the "signFiles" key.

$ ./signJars.sh wso2as-5.2.1/ > log

Configuring WSO2 Product to use Java Security Manager


To configure Java Security Manager, we need to pass few arguments to the main Java process. 

Java Security Manager can be enabled by using "java.security.manager" system property. We will specify the WSO2 Carbon Security Manager using this argument.

We also need to specify the security policy file using "java.security.policy" system property.

As I mentioned earlier, we will also set "restricted.packages" & "denied.system.properties" system properties.

Following is the recommended set of values to be used in wso2server.sh (Edit the startup script and add following lines just before the line " org.wso2.carbon.bootstrap.Bootstrap $*"


    -Djava.security.manager=org.wso2.carbon.bootstrap.CarbonSecurityManager \
    -Djava.security.policy=$CARBON_HOME/repository/conf/sec.policy \
    -Drestricted.packages=sun.,com.sun.xml.internal.ws.,com.sun.xml.internal.bind.,com.sun.imageio.,org.wso2.carbon. \
    -Ddenied.system.properties=javax.net.ssl.trustStore,javax.net.ssl.trustStorePassword,denied.system.properties \


Exporting signFiles public key certificate and importing it to wso2carbon.jks


We need to import the signFiles public key certificate to the wso2carbon.jks as the security policy file will be referring the signFiles signer certificate from the wso2carbon.jks (as specified by the first line).

$ keytool -export -keystore signkeystore.jks -alias signFiles -file sign-cert.cer
$ keytool -import -alias signFiles -file sign-cert.cer -keystore wso2as-5.2.1/repository/resources/security/wso2carbon.jks

Note: wso2carbon.jks' keystore password is "wso2carbon".

The Security Policy File


As specified in the system property "java.security.policy", we will keep the security policy file at $CARBON_HOME/repository/conf/sec.policy

Following policy file should be enough for starting up WSO2 Application Server and deploying a sample JSF & CXF webapps.


keystore "file:${user.dir}/repository/resources/security/wso2carbon.jks", "JKS";

// ========= Carbon Server Permissions ===================================
grant {
    // Allow socket connections for any host
    permission java.net.SocketPermission "*:1-65535", "connect,resolve";

    // Allow to read all properties. Use -Ddenied.system.properties in wso2server.sh to restrict properties
    permission java.util.PropertyPermission "*", "read";
    
    permission java.lang.RuntimePermission "getClassLoader";
    
    // CarbonContext APIs require this permission
    permission java.lang.management.ManagementPermission "control";

    // Required by any component reading XMLs. For example: org.wso2.carbon.databridge.agent.thrift:4.2.1.
    permission java.lang.RuntimePermission "accessClassInPackage.com.sun.xml.internal.bind.v2.runtime.reflect";

    // Required by org.wso2.carbon.ndatasource.core:4.2.0. This is only necessary after adding above permission. 
    permission java.lang.RuntimePermission "accessClassInPackage.com.sun.xml.internal.bind";
};

// ========= Platform signed code permissions ===========================
grant signedBy "signFiles" {
  permission java.security.AllPermission;
};

// ========= Granting permissions to webapps ============================
grant codeBase "file:${carbon.home}/repository/deployment/server/webapps/-" {

    // Required by webapps. For example JSF apps.
    permission java.lang.reflect.ReflectPermission "suppressAccessChecks";

    // Required by webapps. For example JSF apps require this to initialize com.sun.faces.config.ConfigureListener
    permission java.lang.RuntimePermission "setContextClassLoader";

    // Required by webapps to make HttpsURLConnection etc.
    permission java.lang.RuntimePermission "modifyThreadGroup";

    // Required by webapps. For example JSF apps need to invoke annotated methods like @PreDestroy
    permission java.lang.RuntimePermission "accessDeclaredMembers";

    // Required by webapps. For example JSF apps
    permission java.lang.RuntimePermission "accessClassInPackage.org.apache.jasper.compiler";

    // Required by webapps. For example JSF EL
    permission java.lang.RuntimePermission "getClassLoader";

    // Required by CXF app. Needed when invoking services
    permission javax.xml.bind.JAXBPermission "setDatatypeConverter";

    // File reads required by JSF (Sun Mojarra & MyFaces require these)
    // MyFaces has a fix https://issues.apache.org/jira/browse/MYFACES-3590   
    permission java.io.FilePermission "/META-INF", "read";
    permission java.io.FilePermission "/META-INF/-", "read";

    // OSGi permissions are requied to resolve bundles. Required by JSF
    permission org.osgi.framework.AdminPermission "*", "resolve,resource";
};

The security policies may vary depending on your requirements. I recommend to test your application thoroughly in a development environment.

NOTE: There are risks in allowing some Runtime Permissions. Please look at the java docs for RuntimePermission. See Concerns below.

Troubleshooting Java Security


Java provides the "java.security.debug" system property to set various debugging options and monitor security access.

I recommend to add following line to wso2server.sh whenever you need to troubleshoot some issue with Java Security.

    -Djava.security.debug="access,failure"

After adding that line, all the debug information will be printed to standard output. To check the logs, we can start the server using nohup.

$ nohup ./wso2server.sh &

Then we can grep the nohup.out and look for access denied messages.

$ tailf nohup.out | grep denied

Concerns with Java Security Policy


There are few concerns with current permission model in WSO2 products.

  • Use of ManagementPermission instead of Carbon specific permissions. The real ManagementPermission is used for a different purpose. I created CARBON-14966 jira to fix that.
  • Ideally the permission 'java.lang.management.ManagementPermission "control"' should not be specified in policy file as it is only required for privileged actions in Carbon. However due to indirect usage of such privileged actions within Carbon code, we need to specify that permission. This also needs to be fixed.
  • In above policy file, the JSF webapps etc require some risky runtime permissions. I recommend to use a Custom Runtime Environment (CRE) in WSO2 Application Server for JSF webapps etc and sign the jars inside CRE. You can also grant permissions based on the jar names (Use grant codeBase). However signing jars and using a CRE is a better approach with WSO2 AS.
If you also encounter any issues when using Java Security Manager, please discuss those issues in our developer mailing list.




Thursday, November 13, 2014

Java Performance Monitoring Libraries

There is a proposal to build performance probes in WSO2 Platform. For that I started looking in to some performance monitoring libraries.

Following libraries were mentioned in the WSO2 architecture thread.
While looking in to these libraries, I found out about following also.

Here is a quick comparison of each project. These comparison criteria are based on the requirements in above proposal.



Metrics ParfaitJAMonJava SimonPerf4J
LicenseApache License 2.0Apache License 2.0JAMon LicenseNew BSD LicenseApache License 2.0
SourceGitHubGoogle CodeSourceforgeGitHubGitHub
Latest Version3.1.00.2.82.794.0.00.9.16
Last PublishedSep 4, 2014Jun 01, 2011Aug 20, 2014Oct 29, 2014Oct 16, 2011
Java Version-Java 6-Java 7-
CountsYesYesYesYesNo
TimingsYesYesYesYesYes
JMX SupportYesYesNoYesYes
Enable/DisableNoNo*YesYesYes*

* Not confirmed

Let's look at each library in brief.

Metrics


Metrics provides various measuring instruments. 
  • Meters - Measuring rate of events over time
  • Gauges - Instantaneous measurement of a value
  • Counters - Measurement for counting
  • Histograms - Statistical distribution of values
  • Timers - Measures the duration of a code block and the rate of invocation.
  • Health Checks - Centralizing the health checks of services

Metrics has modules for common libraries like Jetty, Logback, Log4j, Apache HttpClient,
Ehcache, JDBI, Jersey.

Metrics provides a way to have multiple reporting options. Mainly JMX, Servlets (http), Console, CSV and SLF4J.  It also supports Ganglia and Graphite reporting

Metrics' Getting Started page shows you how to use the Metrics APIs.

Parfait


Parfait provides mechanisms for collecting counter and timing metrics. Data can be exposed via various mechanisms including JMX and the the open-source cross-platform Performance Co-Pilot.

Parfait also has number of modules, which enable to collect metrics from common data sources.

JAMon


JAMon provides various ways to monitor applications without code changes. It has in built support for HTTP Monitoring, Spring, JDBC, Log4j, EJB etc. JAMon can be used in Servlets, JSPs, EJBs and Java Beans in various Java EE Application Servers.

JAMon doesn't seem to support JMX.

Java Simon


Java Simon has monitors called Simons, which can be used in the code to count something or to measure the time taken.

It's interesting to know that the Java Simon was started by the people, who used JAMon earlier. They were not satisfied with JAMon in terms of simplicity and monitor structure. Some people also consider Java Simon as the replacement of JAMon.

Java Simon also measures time in nanos. Simons are organized in a hierarchy.

Simons can be disabled easily. 

Java Simon has a web console in addition to exposing data via JMX. There are many examples for Java Simon usage and Getting Started wiki is a good place to see how we can use the APIs.

There is comparison of Java Simon with JAMon, which shows performance overhead of each library.

Perf4J


The Perf4J mainly makes use of the logging frameworks and it has support for popular logging frameworks.

The Perf4J support is limited only to timing metrics and I didn't find the support for counters.

Summary


In this blog post, I just wanted to give an idea about Java Performance Monitoring libraries and each library has pros & cons. So, depending on the project requirements, it's better to evaluate all libraries and select a suitable one for your project.

There is lot of information in each project's web pages. Going through those pages will help to understand more about features provided by each library.

Tuesday, June 3, 2014

Monitoring WSO2 products with logstash JMX input plugin

These days, I got the chance to play with ELK (Elasticsearch, Logstash & Kibana). These tools are a great way to analyze & visualize all logs.

You can easily analyze all wso2carbon.log files from ELK. However we also needed to use ELK for monitoring WSO2 products and this post explains the essential steps to use logstash JMX input plugin to monitor WSO2 servers.

Installing Logstash JMX input plugin


Logstash has many inputs and the JMX input plugin is available under "contrib"

We can use "plugin install contrib" command to install extra plugins.

cd /opt/logstash/bin
sudo ./plugin install contrib

Note: If you use logstash 1.4.0 and encounter issues in loading the jmx4r, please refer Troubleshooting below.

Logstash JMX input configuration


When using the JMX input plugin, we can use a similar configuration as follows. We are keeping the logstash configs in "/etc/logstash/conf.d/logstash.conf"


input {
jmx{
    path => "/etc/logstash/jmx"
    polling_frequency => 30
    type => "jmx"
    nb_thread => 4
  }
}

output {
  elasticsearch { host => localhost }
}

Note that the path points to a directory. We have the JMX configuration in "/etc/logstash/jmx/jmx.conf"

{
  //The WSO2 server hostname
  "host" : "localhost",
  //jmx listening port
  "port" : 9999,
  //username to connect to jmx
  "username" : "jmx_user",
  //password to connect to jmx
  "password": "jmx_user_pw",
  "alias" : "jmx.dssworker1.elasticsearch",
  //List of JMX metrics to retrieve
  "queries" : [
    {
      "object_name" : "java.lang:type=Memory",
      "attributes" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
      "object_alias" : "Memory"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=Code Cache",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolCodeCache"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Perm Gen",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolPermGen"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Old Gen",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolOldGen"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Eden Space",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolEdenSpace"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Survivor Space",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolSurvivorSpace"
    }, {
      "object_name" : "java.lang:type=GarbageCollector,name=*MarkSweep",
      "attributes" : [ "Name", "CollectionCount", "CollectionTime" ],
      "object_alias" : "GarbageCollectorMarkSweep"
    }, {
      "object_name" : "java.lang:type=GarbageCollector,name=ParNew",
      "attributes" : [ "Name", "CollectionCount", "CollectionTime" ],
      "object_alias" : "GarbageCollectorParNew"
    }, {
      "object_name" : "java.lang:type=ClassLoading",
      "attributes" : [ "LoadedClassCount", "TotalLoadedClassCount", "UnloadedClassCount" ],
      "object_alias" : "ClassLoading"
    }, {
      "object_name" : "java.lang:type=Runtime",
      "attributes" : [ "Uptime", "StartTime" ],
      "object_alias" : "Runtime"
    }, {
      "object_name" : "java.lang:type=Threading",
      "attributes" : [ "ThreadCount", "TotalStartedThreadCount", "DaemonThreadCount", "PeakThreadCount" ],
      "object_alias" : "Threading"
    }, {
      "object_name" : "java.lang:type=OperatingSystem",
      "attributes" : [ "OpenFileDescriptorCount", "FreePhysicalMemorySize", "CommittedVirtualMemorySize", "FreeSwapSpaceSize", "ProcessCpuLoad", "ProcessCpuTime", "SystemCpuLoad", "TotalPhysicalMemorySize", "TotalSwapSpaceSize", "SystemLoadAverage" ],
      "object_alias" : "OperatingSystem"
    } ]
}


This is all we need to configure logstash to get JMX details from WSO2 servers. Note that we have given a directory as the path for JMX configuration. This means that all the configs inside "/etc/logstash/jmx" will be loaded. So, we need to make sure that there are no other files.

I'm querying only the required attributes for now. It is possible to add as many queries as you need.

Securing JMX access of WSO2 servers.


WSO2 servers by default start the JMX service and you should be able to see the JMX Service URL in wso2carbon.log

For example:
TID: [-1234] [] [DSS] [2014-05-31 01:09:11,103]  INFO {org.wso2.carbon.core.init.JMXServerManager} -  JMX Service URL  : service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi

You can see JMX configuration in <CARBON_HOME>/repository/conf/etc/jmx.xml and the JMX ports in <CARBON_HOME&gt/repository/conf/carbon.xml

       
        <!-- The JMX Ports -->
        <JMX>
            <!--The port RMI registry is exposed-->
            <RMIRegistryPort>9999</RMIRegistryPort>
            <!--The port RMI server should be exposed-->
            <RMIServerPort>11111</RMIServerPort>
        </JMX>

You may change ports from this configuration.

It is recommended to create a role with only "Server Admin" permission and assign to the "jmx_user". Then the "jmx_user" will have the required privileges to monitor WSO2 servers.

Also if we enable Java Security Manager, we need to have following permissions. Usually the WSO2 servers are configured to use the security policy file at  <CARBON_HOME>/repository/conf/sec.policy if the Security Manager is enabled.


grant {
    // JMX monitoring requires following permissions. Check Logstash JMX input configurations
    permission javax.management.MBeanPermission "-#-[-]", "queryNames";
    permission javax.management.MBeanPermission "sun.management.MemoryImpl#*[java.lang:type=Memory]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.MemoryPoolImpl#*[java.lang:type=MemoryPool,name=*]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.GarbageCollectorImpl#*[java.lang:type=GarbageCollector,name=*]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.ClassLoadingImpl#*[java.lang:type=ClassLoading]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.RuntimeImpl#*[java.lang:type=Runtime]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.ThreadImpl#*[java.lang:type=Threading]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#*[java.lang:type=OperatingSystem]", "queryNames,getMBeanInfo,getAttribute";
}

That's it. You should be able to push JMX stats via logstash now.


Troubleshooting

First of all you can check whether the configurations are correct by running following command.

logstash --configtest

This must tell that the configuration is OK.

However I encountered following issue in logstash 1.4.0 when running the logstash command.

LoadError: no such file to load -- jmx4r
     require at org/jruby/RubyKernel.java:1085
     require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
     require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:53
     require at /opt/logstash/lib/logstash/JRUBY-6970.rb:27
     require at /opt/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.4/lib/polyglot.rb:65
  thread_jmx at /opt/logstash/bin/lib//logstash/inputs/jmx.rb:132
         run at /opt/logstash/bin/lib//logstash/inputs/jmx.rb:251


For this issue, we need to extract the plugins to the same directory of logstash installation instead of contrib plugin installation. I got help from #logstash IRC to figure this out. Thanks terroNZ!

I did following steps

wget  --no-check-certificate -O logstash-contrib-1.4.0.tar.gz http://download.elasticsearch.org/logstash/logstash/logstash-contrib-1.4.0.tar.gz
tar -xvf logstash-contrib-1.4.0.tar.gz
sudo rsync -rv --ignore-existing logstash-contrib-1.4.0/* /opt/logstash/

Please note that JMX input plugin works fine in logstash-1.4.1 after installing contrib plugin and above steps are not required.

Then the next issue can occur is when connecting to WSO2 server. Check <CARBON_HOME>/repository/logs/audit.log and see whether the user can connect successfully. If it is not successful, you should check user permissions.

Another issue can be the failure of JMX queries. You can run logstash with "--debug" option and see debug logs.

I noticed following.
{:timestamp=>"2014-05-30T00:09:29.373000+0000", :message=>"Find all objects name java.lang:type=Memory", :level=>:debug, :file=>"logstash/inputs/jmx.rb", :line=>"165"}
{:timestamp=>"2014-05-30T00:09:29.392000+0000", :message=>"No jmx object found for java.lang:type=Memory", :level=>:warn, :file=>"logstash/inputs/jmx.rb", :line=>"221"}
{:timestamp=>"2014-05-30T00:09:29.393000+0000", :message=>"Find all objects name java.lang:type=Runtime", :level=>:debug, :file=>"logstash/inputs/jmx.rb", :line=>"165"}
{:timestamp=>"2014-05-30T00:09:29.396000+0000", :message=>"No jmx object found for java.lang:type=Runtime", :level=>:warn, :file=>"logstash/inputs/jmx.rb", :line=>"221"}


This issue came as we have enabled the Java Security Manager and after adding permissions as mentioned above, the logstash JMX input plugin worked fine.

Next is to create dashboards in Kibana using these data. Hopefully I will be able to write a blog post on that as well.


Thursday, February 27, 2014

"Subversion Native Library Not Available" Error on Eclipse (with Subclipse) Ubuntu

Do you use Eclipse and Subclipse?

Do you encounter following error:

"Subversion Native Library Not Available"


Now if you go to Eclipse -> Preferences -> Team -> SVN, you will see that "JavaHL (JNI) is Not Available" for SVN interface.

The solution is to install JavaHL. After installing, we need to make sure that Eclipse can see the relevant libraries. For that I just created soft links.

sudo apt-get install libsvn-java
sudo ln -s /usr/lib/x86_64-linux-gnu/jni/* /usr/lib/

Now restart Eclipse and you should be able use Subsclipse with JavaHL.



Friday, January 24, 2014

Why you should attend WSO2Con Asia 2014

WSO2Con Asia 2014, a must attend conference for anyone interested in WSO2 products, is just around the corner.

The WSO2Con Asia 2014 is scheduled from March 24 to March 26 with two conference days and one pre-conference tutorial day. Have a look at WSO2Con Asia 2014 Agenda for all interesting talks and tutorials WSO2 has to offer.

The conference will be held at Waters Edge, a popular venue in Sri Jayawardenapura Kotte, Sri Lanka.

Here are few reasons I can think of as to why you should attend the WSO2Con.


  • WSO2 is the only company to have a complete set of open source middleware products built from scratch and WSO2Con is a good opportunity for you to learn how we make world-class products.
  • Pre-conference tutorials will help you to get an understanding of how our stuff works. So, don't worry if you do not have much idea about the WSO2 products.
  • See what WSO2 has to offer with WSO2 Mobile subsidiary.
  • Meet the experts.
  • Networking opportunities.
  • Discover how WSO2 software can impact your daily life.
  • Be inspired. I'm sure you will be able to find many reasons (just as I did from previous conferences)
  • Our long term target is to be the #1 middleware company in the world. This conference is a milestone in that long journey. So, be part of the history! :)

There is a 25% discount for early registrations. Make sure you grab the opportunity while it lasts.

Have fun at the WSO2Con!

Tuesday, January 21, 2014

What is WSO2 Private PaaS?

In this blog post, I'm going to briefly introduce the WSO2 Private PaaS, an upcoming product from WSO2.

WSO2 Private PaaS


Even though I'm writing about this now, the WSO2 Private PaaS term coined just few weeks after we donated WSO2 Stratos to Apache Foundation in June 2013.

Apache Stratos (incubating) and WSO2 Private PaaS


The WSO2 Private PaaS is built on top of the Apache Stratos (incubating) project, which is currently undergoing major architectural changes from the initial 3.0.0-incubating version.

Following are few major changes you can expect in the next Apache Stratos (incubating) release. i.e. 4.0.0-incubating version.


  • Using a message broker for communication among core Stratos components.
  • Auto scaling now analyze real time data from cartridges like in flight request count and load average. Currently this data is analyzed using WSO2 Complex Event Processor
  • Load Balancer is now a cartridge in Stratos
  • New Stratos Manager UI
  • Stratos Manager now has RESTful services
  • Improved CLI

Lakmal has written a great blog post on explaining all of these.

Why Private?


The main reason is that WSO2 Private PaaS will be distributed as a product and anyone will be able to set up an on-premise, enterprise ready PaaS.

So, what do you get in WSO2 Private PaaS?


In addition to core Stratos framework, the WSO2 Private PaaS will get all WSO2 products as cartridges.

This means that with all WSO2 middleware products as cartridges, the WSO2 Private PaaS will provide the most comprehensive PaaS for enterprises. This is also great considering all the advantages you can get from WSO2.

The WSO2 Private PaaS will also have improved billing and metering for enterprise requirements.

WSO2 will officially support following IaaS providers for WSO2 Private PaaS
  • Amazon Elastic Compute Cloud (EC2)
  • OpenStack
  • SUSE Cloud
  • VMware vCloud

Cartridge Architecture


All WSO2 product cartridges will be configured using Puppet, an automation software, which will help us to configure cartridges easily with the use of templates etc. 

The cartridge will contain the Java and an init script to the start the Puppet Agent. With the use of user-data feature provided by the IaaS, we will just pass the puppet master host and the product type to the cartridges. The Cartridge will then configure itself using Puppet.

There is a "Cartridge Agent" inside the cartridge to communication with other Stratos components.

I will write a more detailed blog post with examples once we have a stable release soon.

You can check WSO2 Private PaaS JIRA on the progress.

Thursday, January 16, 2014

Installing Oracle JDK 7 (Java Development Kit) on Ubuntu

There are many posts on this topic if you search on Google. This post just explains the steps I use to install JDK 7 on my laptop.

Download the JDK from Oracle. The latest version as of now is Java SE 7u51.

I'm on 64-bit machine, therefore I downloaded jdk-7u51-linux-x64.tar.gz

It's easy to get the tar.gz package as we just have to extract the JDK.

I usually extract the JDK to /usr/lib/jvm directory.


sudo mkdir -p /usr/lib/jvm
cd /usr/lib/jvm/
sudo tar -xf ~/Software/jdk-7u51-linux-x64.tar.gz
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.7.0_51/bin/javac" 1
sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.7.0_51/bin/java" 1
sudo update-alternatives --install "/usr/lib/mozilla/plugins/libjavaplugin.so" "mozilla-javaplugin.so" "/usr/lib/jvm/jdk1.7.0_51/jre/lib/amd64/libnpjp2.so" 1
sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/lib/jvm/jdk1.7.0_51/bin/javaws" 1


After installing, we should configure each alternative

sudo update-alternatives --config javac
sudo update-alternatives --config java
sudo update-alternatives --config mozilla-javaplugin.so
sudo update-alternatives --config javaws

Now we can configure JAVA_HOME. We can edit ~/.bashrc and add following.
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_51/

That's it! :)

UPDATE:

Please check Oracle Java Installation script for Ubuntu. All above steps are now automated via an installation script: https://github.com/chrishantha/install-java