Metrics in Apache Cassandra 1.2
date: January 30, 2013
Apache Cassandra exposes various kind of internal metrics from client read/write request latencies to ColumnFamily's Bloom filter false positive ratio through JMX. From version 1.1, Cassandra started using Yammer's Metrics library to track new client request metrics. Cassandra 1.2 expands use of the library to other metrics that are already available and put them together into
org.apache.cassandra.metrics JMX domain. For backward compatibility, metrics are still available through the same JMX location as before (in fact, nodetool is still using old interface), but those implementations are (almost) replaced by new metrics.
So what's good about using Metrics library? Metrics library offers some fundamental tools for tracking metrics such as
Timers with ease of use API.
For example, to track client request read/write latency, we use Timer. You just code to start and stop Timer and it calculates all the interesting metrics we need. As you can see in the picture bellow, write latency now shows various percentiles, min, mean, max values and request rates.
One nice feature that Metrics library has is the ability to export metrics to files or other systems periodically.
There are official "metrics reporters" to export metrics to csv files, console, and external systems like Ganglia and Graphite. Also, there are community developed reporters like statsd reporter.
Example: Graphite integration
Let me show you quick example. Metrics reporting feature is not fully integrated into Cassandra yet (there is a ticket for this), so you need to do some coding to configure reporting.
Since we don't want to add code to Cassandra itself, we use java agent to setup metrics reporting to Graphite.
1. Get Metrics Graphite reporter jar
You need to grab metrics-graphite-2.0.3.jar and place it to $CASSANDRA_HOME/lib directory. When Cassandra starts up, it is added to classpath automatically. We also use it to compile our agent.
2. Write java code to configure GraphiteReporter
|public class ReportAgent|
|public static void premain(String agentArgs, Instrumentation inst)|
|// reports metrics to Graphite at 127.0.0.1:2003, every 60 seconds|
|GraphiteReporter.enable(60, TimeUnit.SECONDS, "127.0.0.1", 2003);|
You also need MANIFEST.MF to run as a java agent
3. Package into jar
Compile the code and package into jar file. For example:
$ javac -cp $CASSANDRA_HOME/lib/metrics-core-2.0.3.jar:$CASSANDRA_HOME/lib/metrics-graphite-2.0.3.jar com/datastax/example/ReportAgent.java
$ jar -cfM reporter.jar .
4. Edit cassandra-env.sh
You need to add following line at the end of "conf/cassandra-env.sh" to load your jar as java agent.
5. Start Cassandra
And voilà. Eventually you can see your Metrics through Graphite web interface.
There is a large number of metrics available in Cassandra so I cannot list all up in this post. Instead, you can find the list of new metrics and their brief explanations at Apache Cassandra Wiki.
In this article, I showed how cassandra's metrics are built using Yammer's Metrics library and example reporting using Graphite. New metrics offer much more information than previous versions, and are easily integrated to other systems.