Debugging SSTables in 3.0 with sstabledump
date: March 9, 2016
Cassandra 3.0.4 and 3.4 introduces
sstabledump, a new utility for exploring SSTables.
sstabledump is the spiritual successor to and a replacement for
sstable2json was removed from Cassandra in version 3.0, but examining SSTable data is still a useful diagnostic tool.
sstabledump can export SSTable content to the human readable JSON format.
How SSTable data is stored on disk has changed in Cassandra 3.0, as previously covered in ‘Putting some structure in the storage engine’. Previously, SSTables were composed of partition keys and their cells; now SSTables are composed of partitions and their rows.
This eliminates quite a bit of overhead present in prior versions of Cassandra. Metadata such as clustering key values, timestamps and TTLs are now defined at the row level, rather than repeated for each individual cell within a row. This new layout now matches how data is represented in CQL, and is more understandable.
A nice enhancement of
sstable2json is that the utility can be run in ‘client mode’, so the system data does not have to be read to determine schema.
sstabledump can be executed outside of the Cassandra environment, and cassandra.yaml is not required in the classpath for the tool to work.
sstabledump only supports Cassandra 3.X SSTables.
Visualizing the Storage Engine changes in 3.0
sstabledump and the changes in SSTable layout in 3.0, we’ll use
sstabledump to contrast the SSTables created by a Cassandra 2.2 node and those created by a Cassandra 3.0 node.
First, let’s generate a small SSTable for a table that represents stock ticker data. This should be done within a cqlsh session on each Cassandra cluster:
Next, let’s flush memtables to disk as SSTables using
Then in a cqlsh session we will set a column value to null and delete an entire row to generate some tombstones:
We proceed to flush again to generate a new SSTable, and then perform a major compaction yielding a single SSTable.
Now that we have a single SSTable representing operations on our CQL table we can use the appropriate tool to examine its contents.
C* 2.2 sstable2json Output
As previously stated, the sstable2json output demonstrates that the storage engine prior to Cassandra 2.2 represents partition keys and their cells.
A large portion of the presented data in cells is redundant. For example, when we executed INSERT queries, each cell representing a column value shares the same timestamp and TTL. Additionally, each cell contains not only the full name of the column, but also the values of the clustering keys that cell belongs to. This overhead contributes a large portion to the size of the SSTable.
C* 3.0 sstabledump output
As a consequence of the new tool's verbose output, the output payload is less compact than
sstable2json. However, the enriched structure of the 3.0 storage engine, is displayed. What is apparent is that there is less repeated data, which leads to a dramatically reduced SSTable storage footprint.
Looking at the output, note that clustering, timestamp and ttl information are now presented at the row level, instead of repeating in individual cells. This change is a large factor in optimizing disk space. While column names are present in each cell, the full column names are not stored for each cell as previously. You can read more about these optimizations and others in the aforementioned blog post.
Internal Representation Format
As previously mentioned,
sstabledump’s JSON representation is more verbose than
sstabledump also provides an alternative ‘debug’ output format that is more concise than its json counterpart. While initially difficult to understand, it is a more compact and convenient format for advanced users to grok the contents of an SSTable. To view data in this format, simply pass the
-d parameter to
Other than the inclusion of this internal representation format, the usage between
sstable2json is exactly the same.
- Putting some structure in the storage engine - Sylvain Lebresne, DataStax
- Introduction to the Apache Cassandra 3.X Storage Engine - Aaron Morton, The Last Pickle
- Overview of CASSANDRA-8099 changes
- CASSANDRA-8099 - Refactor and modernize the storage engine
- CASSANDRA-7464 - Replace sstable2json