Advanced request tracing in Cassandra 1.2

By Jonathan Ellis -  November 16, 2012 | 4 Comments

Accessing saved trace data

In my first post on request tracing in Cassandra, you may have noticed some extra output from cqlsh like this:

Tracing session: 4ad36250-1eb4-11e2-0000-fe8ebeead9f9

Cassandra automatically saves trace sessions to the system_traces keyspace for reference. Trace data is expired after 24h; if you want to keep a session result longer than that, you'll need to copy it to a more permanent home.

The system_traces keyspace contains two tables, sessions and events:

CREATE TABLE sessions (
  session_id uuid PRIMARY KEY,
  coordinator inet,
  duration int,
  parameters map,
  request text,
  started_at timestamp

  session_id uuid,
  event_id timeuuid,
  activity text,
  source inet,
  source_elapsed int,
  thread text,
  PRIMARY KEY (session_id, event_id)

Note that cqlsh's rendering of a traced request omits the request parameters as well as the event thread. The parameters are redundant when dealing with an interactive query from cqlsh, but can be useful when probabilistic tracing is enabled (see below).

Also note that activity is a simple text field. Do not rely on this remaining unchanged in future Cassandra releases; it is likely that this will be changed to some kind of enum to make mechanical trace processing easier.

Probabilistic tracing

Besides tracing requests interactively, a coordinator can also be told to trace a proportion of all requests it handles with nodetool settraceprobability. This is useful if your application observes intermittent query slowness, but you're not sure which queries are responsible.

Be judicious with this: tracing a request will usually requre at least 10 rows to be inserted, so it is far from free. Unless you are under very light load tracing all requests (probability 1.0) will probably overwhelm your system. I recommend starting with a small fraction, e.g. 0.001 and increasing that only if necessary.

DataStax has many ways for you to advance in your career and knowledge.

You can take free classes, get certified, or read one of our many white papers.

register for classes

get certified

DBA's Guide to NoSQL


  1. Aravind Dhakshinamoorthy says:

    In the sessions column family, the duration is mentioned as int. Does this parameter represents the execution time for the particular request in micro seconds (or milliseconds) ?


    1. Don Neufeld says:

      It’s in microseconds, per the source code

  2. Pawan says:

    when I enable Tracing ON on my Cassandra DB and perform an insert operation through the NodeJS application, I dont see any tracing logged in the system_traces tables. However if I perform insert/select operation in the cqlsh, i am able to see the tracing details. Am I doing anything wrong?

    1. Tyler Hobbs Tyler Hobbs says:

      Running “TRACING ON” in cqlsh only enables tracing for queries through cqlsh. The drivers have their own options for tracing. I would check the docs for the nodejs driver on how to enable tracing for a query.


Your email address will not be published. Required fields are marked *

Subscribe for newsletter:

Tel. +1 (408) 933-3120 Offices France GermanyJapan

DataStax Enterprise is powered by the best distribution of Apache Cassandra™.

© 2017 DataStax, All Rights Reserved. DataStax, Titan, and TitanDB are registered trademark of DataStax, Inc. and its subsidiaries in the United States and/or other countries.
Apache Cassandra, Apache, Tomcat, Lucene, Solr, Hadoop, Spark, TinkerPop, and Cassandra are trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.