Jonathan Ellis

<h3>Accessing saved trace data</h3>

In my first post on&nbsp;<a href="https://www.datastax.com/dev/blog/tracing-in-cassandra-1-2">request tracing in Cassandra</a>, you may have noticed some extra output from&nbsp;<tt>cqlsh</tt>&nbsp;like this:

&nbsp;

<pre>
<code>Tracing session: 4ad36250-1eb4-11e2-0000-fe8ebeead9f9
</code></pre>

&nbsp;

Cassandra automatically saves trace sessions to the&nbsp;<tt>system_traces</tt>&nbsp;keyspace for reference. Trace data is expired after 24h; if you want to keep a session result longer than that, you'll need to copy it to a more permanent home.

The&nbsp;<tt>system_traces</tt>&nbsp;keyspace contains two tables,&nbsp;<tt>sessions</tt>&nbsp;and&nbsp;<tt>events</tt>:

&nbsp;

<pre>
<code>CREATE TABLE sessions (
 session_id uuid PRIMARY KEY,
 coordinator inet,
 duration int,
 parameters map,
 request text,
 started_at timestamp
);

CREATE TABLE events (
 session_id uuid,
 event_id timeuuid,
 activity text,
 source inet,
 source_elapsed int,
 thread text,
 PRIMARY KEY (session_id, event_id)
);
</code></pre>

&nbsp;

Note that cqlsh's rendering of a traced request omits the request&nbsp;<tt>parameters</tt>&nbsp;as well as the event&nbsp;<tt>thread</tt>. The parameters are redundant when dealing with an interactive query from cqlsh, but can be useful when probabilistic tracing is enabled (see below).

Also note that&nbsp;<tt>activity</tt>&nbsp;is a simple text field. Do not rely on this remaining unchanged in future Cassandra releases; it is likely that this will be changed to some kind of enum to make mechanical trace processing easier.

<h3>Probabilistic tracing</h3>

Besides tracing requests interactively, a coordinator can also be told to trace a proportion of all requests it handles with&nbsp;<tt>nodetool settraceprobability</tt>. This is useful if your application observes intermittent query slowness, but you're not sure which queries are responsible.

Be judicious with this: tracing a request will usually requre at least 10 rows to be inserted, so it is far from free. Unless you are under very light load tracing all requests (probability 1.0) will probably overwhelm your system. I recommend starting with a small fraction, e.g. 0.001 and increasing that only if necessary.

Advanced request tracing in Cassandra 1.2

Jonathan EllisTechnology

Share

More Technology

Why You Can't Miss Cassandra Summit

Mitigate the OpenAI Debacle with Open Source and Cloud

Supercharge AI Assistants with Superagent and DataStax

DataStax and AWS: Accelerating GenAI Applications into Production

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale