Metrics in Apache Cassandra 1.2

By Yuki Morishita -  January 30, 2013 | 7 Comments

Apache Cassandra exposes various kind of internal metrics from client read/write request latencies to ColumnFamily's Bloom filter false positive ratio through JMX. From version 1.1, Cassandra started using Yammer's Metrics library to track new client request metrics. Cassandra 1.2 expands use of the library to other metrics that are already available and put them together into org.apache.cassandra.metrics JMX domain. For backward compatibility, metrics are still available through the same JMX location as before (in fact, nodetool is still using old interface), but those implementations are (almost) replaced by new metrics.

Motivation

So what's good about using Metrics library? Metrics library offers some fundamental tools for tracking metrics such as Counters, Gauges and Timers with ease of use API.

For example, to track client request read/write latency, we use Timer. You just code to start and stop Timer and it calculates all the interesting metrics we need. As you can see in the picture bellow, write latency now shows various percentiles, min, mean, max values and request rates.

jconsole showing write latency metrics

Exporting metrics

One nice feature that Metrics library has is the ability to export metrics to files or other systems periodically.
There are official "metrics reporters" to export metrics to csv files, console, and external systems like Ganglia and Graphite. Also, there are community developed reporters like statsd reporter.

Example: Graphite integration

Let me show you quick example. Metrics reporting feature is not fully integrated into Cassandra yet (there is a ticket for this), so you need to do some coding to configure reporting.
Since we don't want to add code to Cassandra itself, we use java agent to setup metrics reporting to Graphite.

1. Get Metrics Graphite reporter jar

You need to grab metrics-graphite-2.0.3.jar and place it to $CASSANDRA_HOME/lib directory. When Cassandra starts up, it is added to classpath automatically. We also use it to compile our agent.

2. Write java code to configure GraphiteReporter

You also need MANIFEST.MF to run as a java agent

3. Package into jar

Compile the code and package into jar file. For example:

$ javac -cp $CASSANDRA_HOME/lib/metrics-core-2.0.3.jar:$CASSANDRA_HOME/lib/metrics-graphite-2.0.3.jar com/datastax/example/ReportAgent.java
$ jar -cfM reporter.jar .

4. Edit cassandra-env.sh

You need to add following line at the end of "conf/cassandra-env.sh" to load your jar as java agent.

JVM_OPTS="-javaagent:/path/to/your/reporter.jar $JVM_OPTS"

5. Start Cassandra

And voilà. Eventually you can see your Metrics through Graphite web interface.

Graphite web interface showing Cassandra metrics

Available metrics

There is a large number of metrics available in Cassandra so I cannot list all up in this post. Instead, you can find the list of new metrics and their brief explanations at Apache Cassandra Wiki.

Conclusion

In this article, I showed how cassandra's metrics are built using Yammer's Metrics library and example reporting using Graphite. New metrics offer much more information than previous versions, and are easily integrated to other systems.









DataStax has many ways for you to advance in your career and knowledge.

You can take free classes, get certified, or read one of our many white papers.



register for classes

get certified

DBA's Guide to NoSQL







Comments

  1. sanjay says:

    Can anyone give me a sample Report agent code.

    Thanks

  2. ganga says:

    Hi Yuki, We could able to pull the metrics using the agent reporter which based on yammer metrics library in DSE cassandra version 4.8.0, but not in 5.1.0+. Is something broken or removed support for yammer metrics library ?

    1. Yuki Morishita says:

      There were update on library and deprecation of metrics since this article was written (this article was for v1.2).

      Nowadays you can just write metrics reporting config and put reporter jar file for external reporting. Cassandra uses uses https://github.com/addthis/metrics-reporter-config to config reporting.

      See conf/metrics-reporter-config-sample.yaml file under your cassandra directory.

      1. ganga says:

        Thanks Yuki for the quick response . This is very helpful . Do you have the sample for reporting agent code to emit the metrics for external metrics? I am trying to emit the metrics to UDP port using StatsD to consume our coustom metric system.

        1. ganga says:

          Typo “external metrics ” to “external reporting”

          1. ganga says:

            Hi Yuki,
            I am able to get custom reporter (jar) working and report method on my customer reporter being invoked by scheduler reporter, but all the metrics are empty even though I passed the MetricFilter.ALL for filter value. Do you know what I am missing here? Btw, I have not used “metrics-reporter-config” which I believe is optional. Please correct me if I am wrong. Appreciate some pointers on resolving this issue.

  3. ganga says:

    Hi Yuki,

    Thanks for pointers. Able to get all the metrics except the CASRead,CASWrite, Read and Write metrics related. Performed reads and writes via dev center, but these metrics are emitted checked through ConsoleReporter. Does these metrics require extra configuration?

Comments

Your email address will not be published. Required fields are marked *




Subscribe for newsletter:

Tel. +1 (408) 933-3120 sales@datastax.com Offices France GermanyJapan

DataStax Enterprise is powered by the best distribution of Apache Cassandra™.

© 2017 DataStax, All Rights Reserved. DataStax, Titan, and TitanDB are registered trademark of DataStax, Inc. and its subsidiaries in the United States and/or other countries.
Apache Cassandra, Apache, Tomcat, Lucene, Solr, Hadoop, Spark, TinkerPop, and Cassandra are trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.