Monitoring WSO2 products with logstash JMX input plugin

These days, I got the chance to play with ELK (Elasticsearch, Logstash & Kibana). These tools are a great way to analyze & visualize all logs.

You can easily analyze all wso2carbon.log files from ELK. However we also needed to use ELK for monitoring WSO2 products and this post explains the essential steps to use logstash JMX input plugin to monitor WSO2 servers.

Installing Logstash JMX input plugin


Logstash has many inputs and the JMX input plugin is available under "contrib"

We can use "plugin install contrib" command to install extra plugins.

cd /opt/logstash/bin
sudo ./plugin install contrib

Note: If you use logstash 1.4.0 and encounter issues in loading the jmx4r, please refer Troubleshooting below.

Logstash JMX input configuration


When using the JMX input plugin, we can use a similar configuration as follows. We are keeping the logstash configs in "/etc/logstash/conf.d/logstash.conf"


input {
jmx{
    path => "/etc/logstash/jmx"
    polling_frequency => 30
    type => "jmx"
    nb_thread => 4
  }
}

output {
  elasticsearch { host => localhost }
}

Note that the path points to a directory. We have the JMX configuration in "/etc/logstash/jmx/jmx.conf"

{
  //The WSO2 server hostname
  "host" : "localhost",
  //jmx listening port
  "port" : 9999,
  //username to connect to jmx
  "username" : "jmx_user",
  //password to connect to jmx
  "password": "jmx_user_pw",
  "alias" : "jmx.dssworker1.elasticsearch",
  //List of JMX metrics to retrieve
  "queries" : [
    {
      "object_name" : "java.lang:type=Memory",
      "attributes" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
      "object_alias" : "Memory"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=Code Cache",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolCodeCache"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Perm Gen",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolPermGen"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Old Gen",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolOldGen"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Eden Space",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolEdenSpace"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Survivor Space",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolSurvivorSpace"
    }, {
      "object_name" : "java.lang:type=GarbageCollector,name=*MarkSweep",
      "attributes" : [ "Name", "CollectionCount", "CollectionTime" ],
      "object_alias" : "GarbageCollectorMarkSweep"
    }, {
      "object_name" : "java.lang:type=GarbageCollector,name=ParNew",
      "attributes" : [ "Name", "CollectionCount", "CollectionTime" ],
      "object_alias" : "GarbageCollectorParNew"
    }, {
      "object_name" : "java.lang:type=ClassLoading",
      "attributes" : [ "LoadedClassCount", "TotalLoadedClassCount", "UnloadedClassCount" ],
      "object_alias" : "ClassLoading"
    }, {
      "object_name" : "java.lang:type=Runtime",
      "attributes" : [ "Uptime", "StartTime" ],
      "object_alias" : "Runtime"
    }, {
      "object_name" : "java.lang:type=Threading",
      "attributes" : [ "ThreadCount", "TotalStartedThreadCount", "DaemonThreadCount", "PeakThreadCount" ],
      "object_alias" : "Threading"
    }, {
      "object_name" : "java.lang:type=OperatingSystem",
      "attributes" : [ "OpenFileDescriptorCount", "FreePhysicalMemorySize", "CommittedVirtualMemorySize", "FreeSwapSpaceSize", "ProcessCpuLoad", "ProcessCpuTime", "SystemCpuLoad", "TotalPhysicalMemorySize", "TotalSwapSpaceSize", "SystemLoadAverage" ],
      "object_alias" : "OperatingSystem"
    } ]
}


This is all we need to configure logstash to get JMX details from WSO2 servers. Note that we have given a directory as the path for JMX configuration. This means that all the configs inside "/etc/logstash/jmx" will be loaded. So, we need to make sure that there are no other files.

I'm querying only the required attributes for now. It is possible to add as many queries as you need.

Securing JMX access of WSO2 servers.


WSO2 servers by default start the JMX service and you should be able to see the JMX Service URL in wso2carbon.log

For example:
TID: [-1234] [] [DSS] [2014-05-31 01:09:11,103]  INFO {org.wso2.carbon.core.init.JMXServerManager} -  JMX Service URL  : service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi

You can see JMX configuration in <CARBON_HOME>/repository/conf/etc/jmx.xml and the JMX ports in <CARBON_HOME&gt/repository/conf/carbon.xml

       
        <!-- The JMX Ports -->
        <JMX>
            <!--The port RMI registry is exposed-->
            <RMIRegistryPort>9999</RMIRegistryPort>
            <!--The port RMI server should be exposed-->
            <RMIServerPort>11111</RMIServerPort>
        </JMX>

You may change ports from this configuration.

It is recommended to create a role with only "Server Admin" permission and assign to the "jmx_user". Then the "jmx_user" will have the required privileges to monitor WSO2 servers.

Also if we enable Java Security Manager, we need to have following permissions. Usually the WSO2 servers are configured to use the security policy file at  <CARBON_HOME>/repository/conf/sec.policy if the Security Manager is enabled.


grant {
    // JMX monitoring requires following permissions. Check Logstash JMX input configurations
    permission javax.management.MBeanPermission "-#-[-]", "queryNames";
    permission javax.management.MBeanPermission "sun.management.MemoryImpl#*[java.lang:type=Memory]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.MemoryPoolImpl#*[java.lang:type=MemoryPool,name=*]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.GarbageCollectorImpl#*[java.lang:type=GarbageCollector,name=*]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.ClassLoadingImpl#*[java.lang:type=ClassLoading]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.RuntimeImpl#*[java.lang:type=Runtime]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.ThreadImpl#*[java.lang:type=Threading]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#*[java.lang:type=OperatingSystem]", "queryNames,getMBeanInfo,getAttribute";
}

That's it. You should be able to push JMX stats via logstash now.


Troubleshooting

First of all you can check whether the configurations are correct by running following command.

logstash --configtest

This must tell that the configuration is OK.

However I encountered following issue in logstash 1.4.0 when running the logstash command.

LoadError: no such file to load -- jmx4r
     require at org/jruby/RubyKernel.java:1085
     require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
     require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:53
     require at /opt/logstash/lib/logstash/JRUBY-6970.rb:27
     require at /opt/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.4/lib/polyglot.rb:65
  thread_jmx at /opt/logstash/bin/lib//logstash/inputs/jmx.rb:132
         run at /opt/logstash/bin/lib//logstash/inputs/jmx.rb:251


For this issue, we need to extract the plugins to the same directory of logstash installation instead of contrib plugin installation. I got help from #logstash IRC to figure this out. Thanks terroNZ!

I did following steps

wget  --no-check-certificate -O logstash-contrib-1.4.0.tar.gz http://download.elasticsearch.org/logstash/logstash/logstash-contrib-1.4.0.tar.gz
tar -xvf logstash-contrib-1.4.0.tar.gz
sudo rsync -rv --ignore-existing logstash-contrib-1.4.0/* /opt/logstash/

Please note that JMX input plugin works fine in logstash-1.4.1 after installing contrib plugin and above steps are not required.

Then the next issue can occur is when connecting to WSO2 server. Check <CARBON_HOME>/repository/logs/audit.log and see whether the user can connect successfully. If it is not successful, you should check user permissions.

Another issue can be the failure of JMX queries. You can run logstash with "--debug" option and see debug logs.

I noticed following.
{:timestamp=>"2014-05-30T00:09:29.373000+0000", :message=>"Find all objects name java.lang:type=Memory", :level=>:debug, :file=>"logstash/inputs/jmx.rb", :line=>"165"}
{:timestamp=>"2014-05-30T00:09:29.392000+0000", :message=>"No jmx object found for java.lang:type=Memory", :level=>:warn, :file=>"logstash/inputs/jmx.rb", :line=>"221"}
{:timestamp=>"2014-05-30T00:09:29.393000+0000", :message=>"Find all objects name java.lang:type=Runtime", :level=>:debug, :file=>"logstash/inputs/jmx.rb", :line=>"165"}
{:timestamp=>"2014-05-30T00:09:29.396000+0000", :message=>"No jmx object found for java.lang:type=Runtime", :level=>:warn, :file=>"logstash/inputs/jmx.rb", :line=>"221"}


This issue came as we have enabled the Java Security Manager and after adding permissions as mentioned above, the logstash JMX input plugin worked fine.

Next is to create dashboards in Kibana using these data. Hopefully I will be able to write a blog post on that as well.


Location: Seattle, WA, USA

Comments

Post a Comment

Popular posts from this blog

Specifying a custom Event Settings file for Java Flight Recorder

Image
When you are using Java Flight Recorder (JFR), the JFR will use an event settings file to check which event types to record. By default in JFR, there are two settings, "default" and "profile". The default setting is recommended for Continuous Recordings as it has very low overhead (typically less than 1% overhead). The profile setting has more events and useful when profiling the application. As mentioned in my previous blog post regarding  Java Flight Recorder Continuous Recordings , we use following arguments to do a Continuous Recording. -XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:FlightRecorderOptions=defaultrecording=true,disk=true,repository=./tmp,dumponexit=true,dumponexitpath=./ Note: According to the Oracle documentation on " java " command, we should be able to specify "settings" parameter to the -XX:FlightRecorderOptions. However, the settings parameter has no effect when used with the -XX:FlightRecorderOptions and ...
Read more

Flame Graphs with Java Flight Recordings

Flame Graphs Brendon D. Gregg , who is a computer performance analyst , has created  Flame Graphs to visualize stack traces in an interactive way. You must watch his talk at USENIX/LISA13 , titled Blazing Performance with Flame Graphs , which explains Flame Graphs in detail. There can be different types of flame graphs and I'm focusing on  CPU Flame Graphs  with Java in this blog post. Please look at the Flame Graphs Description  to understand the Flame Graph visualization. CPU Flame Graphs and Java Stack Traces As  Brendon  mentioned in his talk, understanding why CPUs are busy is very important when analyzing performance.  CPU Flame Graphs  is a good way to identify hot methods from sampled stack traces. In order to generate CPU Flame Graphs for Java Stack Traces , we need a way to get sample stack traces. Brendon has given examples to use jstack  and Google's lightweight-java-profiler . Please r...
Read more

Java Flight Recorder Continuous Recordings

When we are trying to find performance issues, it is sometimes necessary to do continuous recordings with Java Flight Recorder. Usually we debug issues in an environment similar to a production setup. That means we don't have a desktop environment and we cannot use Java Mission Control for flight recording. That also means we need to record & get dumps using command line in servers. We can of course use remote connection methods, but it's more easier to get recordings from the server. With continuous recordings, we need to figure out how to get dumps. There are few options. Get a dump when the Java application exits. For this, we need to use dumponexit and dumponexitpath options. Get a dump manually from JFR.dump diagnostic command via " jcmd " Note: The "jcmd" command is in $JAVA_HOME/bin. If you use the  Oracle Java Installation script for Ubuntu , you can directly use "jcmd" without including  $JAVA_HOME/bin in $PATH. Enabl...
Read more