CQL for Cassandra 2.x

Tracing consistency changes

In a distributed system such as Cassandra, the most recent value of data is not necessarily on every node all the time. The client application configures the consistency level per request to manage response time versus data accuracy. By tracing activity on a five-node cluster, this tutorial shows the difference between these consistency levels and the number of replicas that participate to satisfy a request:
  • ONE

    Returns data from the nearest replica.

  • QUORUM

    Returns the most recent data from the majority of replicas.

  • ALL

    Returns the most recent data from all replicas.

Follow instructions to setup five nodes on your local computer, trace reads at different consistency levels, and then compare the results.

Setup to trace consistency changes

To setup five nodes on your local computer, trace reads at different consistency levels, and then compare the results.

Procedure

  1. Get the ccm library of scripts from github.
    You will use this library in subsequent steps to perform the following actions:
    • Download Apache Cassandra source code.
    • Create and launch an Apache Cassandra cluster on a single computer.
    Refer to the ccm README for prerequisites.
  2. Set up loopback aliases. For example, enter the following commands on the command line to set up the alias on the Mac. On some platforms, you can probably skip this step.
    $ sudo ifconfig lo0 alias 127.0.0.2 up
    $ sudo ifconfig lo0 alias 127.0.0.3 up
    $ sudo ifconfig lo0 alias 127.0.0.4 up
    $ sudo ifconfig lo0 alias 127.0.0.5 up
  3. Download Apache Cassandra source code, version 2.1.0 for example, into the /.ccm/repository, and start the ccm cluster named trace_consistency.
    $ ccm create trace_consistency -v 2.1.0
    Downloading http://archive.apache.org/dist/cassandra/2.1.0/apache-cassandra-2.1.0-src.tar.gz to /var/folders/9k/ywsprd8n14s7hzb5qnztgb5h0000gq/T/ccm-d7fGAN.tar.gz (15.750MB)
      16514874  [100.00%]
    Extracting /var/folders/9k/ywsprd8n14s7hzb5qnztgb5h0000gq/T/ccm-d7fGAN.tar.gz as version 2.1.0 ...
    Compiling Cassandra 2.1.0 ...
    Current cluster is now: trace_consistency
  4. Use the following commands to populate and check the cluster:
    $ ccm populate -n 5
    $ ccm start
  5. Check that the cluster is up:
    $ ccm node1 ring
    The output shows the status of all five nodes.

Trace reads at different consistency levels

After performing the setup steps, run and trace queries that read data at different consistency levels. The tracing output shows that using three replicas on a five-node cluster, a consistency level of ONE processes responses from one of three replicas, QUORUM from two of three replicas, and ALL from three of three replicas.

Procedure

  1. Connect cqlsh to the first node in the ring.
    $ ccm node1 cqlsh
  2. On the cqlsh command line, create a keyspace that specifies using three replica for data distribution in the cluster.
    cqlsh> CREATE KEYSPACE demo_cl WITH replication = {'class':'SimpleStrategy', 'replication_factor':3};
  3. In the three-replica keyspace, create a table, and insert some values:
    cqlsh> USE demo_cl;
    cqlsh:demo_cl> CREATE TABLE demo_table ( id int PRIMARY KEY, col1 int, col2 int );
    cqlsh:demo_cl> INSERT INTO demo_table (id, col1, col2) VALUES (0, 0, 0);
  4. Turn on tracing and use the CONSISTENCY command to check that the consistency level is ONE, the default.
    cqlsh:demo_cl> TRACING on;
    cqlsh:demo_cl> CONSISTENCY;
    
    The output should be:
    Current consistency level is 1.
  5. Query the table to read the value of the primary key.
    cqlsh:demo_cl> SELECT * FROM demo_table WHERE id = 0;
    The output includes tracing information:
     id | col1 | col2
    ----+------+------
      0 |    0 |    0
    
    (1 rows)
    
    Tracing session: 0f5058d0-6761-11e4-96a3-fd07420471ed
    
     activity                                                                                        | timestamp                  | source    | source_elapsed
    -------------------------------------------------------------------------------------------------+----------------------------+-----------+----------------
                                                                                  Execute CQL3 query | 2014-11-08 08:05:29.437000 | 127.0.0.1 |              0
                    Parsing SELECT * FROM demo_table WHERE id = 0 LIMIT 10000; [SharedPool-Worker-1] | 2014-11-08 08:05:29.438000 | 127.0.0.1 |            820
                                                           Preparing statement [SharedPool-Worker-1] | 2014-11-08 08:05:29.438000 | 127.0.0.1 |           1637
                                                    Sending message to /127.0.0.3 [WRITE-/127.0.0.3] | 2014-11-08 08:05:29.439000 | 127.0.0.1 |           2211
                                                    Sending message to /127.0.0.4 [WRITE-/127.0.0.4] | 2014-11-08 08:05:29.439000 | 127.0.0.1 |           2237
                                                        Message received from /127.0.0.1 [Thread-10] | 2014-11-08 08:05:29.441000 | 127.0.0.3 |             75
                                Executing single-partition query on demo_table [SharedPool-Worker-1] | 2014-11-08 08:05:29.441000 | 127.0.0.4 |            818
                                                  Acquiring sstable references [SharedPool-Worker-1] | 2014-11-08 08:05:29.441000 | 127.0.0.4 |            861
                                                   Merging memtable tombstones [SharedPool-Worker-1] | 2014-11-08 08:05:29.441000 | 127.0.0.4 |            915
     Skipped 0/0 non-slice-intersecting sstables, included 0 due to tombstones [SharedPool-Worker-1] | 2014-11-08 08:05:29.442000 | 127.0.0.4 |            999
                                    Merging data from memtables and 0 sstables [SharedPool-Worker-1] | 2014-11-08 08:05:29.442000 | 127.0.0.4 |           1018
                                                   Merging memtable tombstones [SharedPool-Worker-1] | 2014-11-08 08:05:29.442000 | 127.0.0.3 |           1058
     Skipped 0/0 non-slice-intersecting sstables, included 0 due to tombstones [SharedPool-Worker-1] | 2014-11-08 08:05:29.442000 | 127.0.0.3 |           1146
                                    Merging data from memtables and 0 sstables [SharedPool-Worker-1] | 2014-11-08 08:05:29.442000 | 127.0.0.3 |           1165
                                            Read 1 live and 0 tombstoned cells [SharedPool-Worker-1] | 2014-11-08 08:05:29.442000 | 127.0.0.3 |           1223
                                              Enqueuing response to /127.0.0.1 [SharedPool-Worker-1] | 2014-11-08 08:05:29.442001 | 127.0.0.3 |           1504
                                                         Message received from /127.0.0.4 [Thread-7] | 2014-11-08 08:05:29.443000 | 127.0.0.1 |           6399
                                                    Sending message to /127.0.0.1 [WRITE-/127.0.0.1] | 2014-11-08 08:05:29.443000 | 127.0.0.3 |           1835
                                                         Message received from /127.0.0.3 [Thread-8] | 2014-11-08 08:05:29.443000 | 127.0.0.1 |           6449
                                           Processing response from /127.0.0.4 [SharedPool-Worker-2] | 2014-11-08 08:05:29.443000 | 127.0.0.1 |           6623
                                           Processing response from /127.0.0.3 [SharedPool-Worker-3] | 2014-11-08 08:05:29.443000 | 127.0.0.1 |           6635
                                                                                    Request complete | 2014-11-08 08:05:29.443897 | 127.0.0.1 |           6897
  6. Change the consistency level to QUORUM and run the SELECT statement again.
    cqlsh:demo_cl> CONSISTENCY quorum;
    cqlsh:demo_cl> SELECT * FROM demo_table WHERE id = 0;
     id | col1 | col2
    ----+------+------
      0 |    0 |    0
    
    (1 rows)
    
    
    Tracing session: 3bbae430-6761-11e4-96a3-fd07420471ed
    
     activity                                                                                        | timestamp                  | source    | source_elapsed
    -------------------------------------------------------------------------------------------------+----------------------------+-----------+----------------
                                                                                  Execute CQL3 query | 2014-11-08 08:06:43.955000 | 127.0.0.1 |              0
                    Parsing SELECT * FROM demo_table WHERE id = 0 LIMIT 10000; [SharedPool-Worker-1] | 2014-11-08 08:06:43.955000 | 127.0.0.1 |             71
                                                           Preparing statement [SharedPool-Worker-1] | 2014-11-08 08:06:43.955000 | 127.0.0.1 |            267
                                                    Sending message to /127.0.0.4 [WRITE-/127.0.0.4] | 2014-11-08 08:06:43.956000 | 127.0.0.1 |           1628
                                                    Sending message to /127.0.0.5 [WRITE-/127.0.0.5] | 2014-11-08 08:06:43.956000 | 127.0.0.1 |           1690
                                                         Message received from /127.0.0.1 [Thread-9] | 2014-11-08 08:06:43.957000 | 127.0.0.5 |             95
                                Executing single-partition query on demo_table [SharedPool-Worker-2] | 2014-11-08 08:06:43.957000 | 127.0.0.4 |            229
                                                  Acquiring sstable references [SharedPool-Worker-2] | 2014-11-08 08:06:43.957000 | 127.0.0.4 |            249
                                                   Merging memtable tombstones [SharedPool-Worker-2] | 2014-11-08 08:06:43.957000 | 127.0.0.4 |            299
     Skipped 0/0 non-slice-intersecting sstables, included 0 due to tombstones [SharedPool-Worker-2] | 2014-11-08 08:06:43.957000 | 127.0.0.4 |            387
                                    Merging data from memtables and 0 sstables [SharedPool-Worker-2] | 2014-11-08 08:06:43.957000 | 127.0.0.4 |            408
                                            Read 1 live and 0 tombstoned cells [SharedPool-Worker-2] | 2014-11-08 08:06:43.957000 | 127.0.0.4 |            469
                                              Enqueuing response to /127.0.0.1 [SharedPool-Worker-2] | 2014-11-08 08:06:43.957001 | 127.0.0.4 |            734
                                                    Sending message to /127.0.0.1 [WRITE-/127.0.0.1] | 2014-11-08 08:06:43.957001 | 127.0.0.4 |            894
                                                         Message received from /127.0.0.4 [Thread-7] | 2014-11-08 08:06:43.958000 | 127.0.0.1 |           3383
                                           Processing response from /127.0.0.4 [SharedPool-Worker-2] | 2014-11-08 08:06:43.958000 | 127.0.0.1 |           3612
                                Executing single-partition query on demo_table [SharedPool-Worker-1] | 2014-11-08 08:06:43.959000 | 127.0.0.5 |           1462
                                                  Acquiring sstable references [SharedPool-Worker-1] | 2014-11-08 08:06:43.959000 | 127.0.0.5 |           1509
                                                   Merging memtable tombstones [SharedPool-Worker-1] | 2014-11-08 08:06:43.959000 | 127.0.0.5 |           1569
     Skipped 0/0 non-slice-intersecting sstables, included 0 due to tombstones [SharedPool-Worker-1] | 2014-11-08 08:06:43.959000 | 127.0.0.5 |           1662
                                    Merging data from memtables and 0 sstables [SharedPool-Worker-1] | 2014-11-08 08:06:43.959000 | 127.0.0.5 |           1681
                                            Read 1 live and 0 tombstoned cells [SharedPool-Worker-1] | 2014-11-08 08:06:43.959000 | 127.0.0.5 |           1760
                                              Enqueuing response to /127.0.0.1 [SharedPool-Worker-1] | 2014-11-08 08:06:43.959001 | 127.0.0.5 |           2104
                                                        Message received from /127.0.0.5 [Thread-10] | 2014-11-08 08:06:43.960000 | 127.0.0.1 |           5330
                                                    Sending message to /127.0.0.1 [WRITE-/127.0.0.1] | 2014-11-08 08:06:43.960000 | 127.0.0.5 |           2423
                                           Processing response from /127.0.0.5 [SharedPool-Worker-2] | 2014-11-08 08:06:43.960000 | 127.0.0.1 |           5519
                                                                                    Request complete | 2014-11-08 08:06:43.960947 | 127.0.0.1 |           5947
  7. Change the consistency level to ALL and run the SELECT statement again.
    cqlsh:demo_cl> CONSISTENCY ALL;
    cqlsh:demo_cl> SELECT * FROM demo_table WHERE id = 0;
     id | col1 | col2
    ----+------+------
      0 |    0 |    0
    
    (1 rows)
    
    
    Tracing session: 4da75ca0-6761-11e4-96a3-fd07420471ed
    
     activity                                                                                        | timestamp                  | source    | source_elapsed
    -------------------------------------------------------------------------------------------------+----------------------------+-----------+----------------
                                                                                  Execute CQL3 query | 2014-11-08 08:07:14.026000 | 127.0.0.1 |              0
                    Parsing SELECT * FROM demo_table WHERE id = 0 LIMIT 10000; [SharedPool-Worker-1] | 2014-11-08 08:07:14.026000 | 127.0.0.1 |             73
                                                           Preparing statement [SharedPool-Worker-1] | 2014-11-08 08:07:14.026000 | 127.0.0.1 |            271
                                                    Sending message to /127.0.0.4 [WRITE-/127.0.0.4] | 2014-11-08 08:07:14.027000 | 127.0.0.1 |            978
                                                         Message received from /127.0.0.1 [Thread-9] | 2014-11-08 08:07:14.027000 | 127.0.0.5 |             56
                                                    Sending message to /127.0.0.5 [WRITE-/127.0.0.5] | 2014-11-08 08:07:14.027000 | 127.0.0.1 |           1012
                                Executing single-partition query on demo_table [SharedPool-Worker-2] | 2014-11-08 08:07:14.027000 | 127.0.0.3 |            253
                                                    Sending message to /127.0.0.3 [WRITE-/127.0.0.3] | 2014-11-08 08:07:14.027000 | 127.0.0.1 |           1054
                                                  Acquiring sstable references [SharedPool-Worker-2] | 2014-11-08 08:07:14.027000 | 127.0.0.3 |            275
                                                   Merging memtable tombstones [SharedPool-Worker-2] | 2014-11-08 08:07:14.027000 | 127.0.0.3 |            344
     Skipped 0/0 non-slice-intersecting sstables, included 0 due to tombstones [SharedPool-Worker-2] | 2014-11-08 08:07:14.028000 | 127.0.0.3 |            438
                                                  Acquiring sstable references [SharedPool-Worker-1] | 2014-11-08 08:07:14.028000 | 127.0.0.5 |            461
                                                   Merging memtable tombstones [SharedPool-Worker-1] | 2014-11-08 08:07:14.028000 | 127.0.0.5 |            525
     Skipped 0/0 non-slice-intersecting sstables, included 0 due to tombstones [SharedPool-Worker-1] | 2014-11-08 08:07:14.028000 | 127.0.0.5 |            622
                                    Merging data from memtables and 0 sstables [SharedPool-Worker-1] | 2014-11-08 08:07:14.028000 | 127.0.0.5 |            645
                                            Read 1 live and 0 tombstoned cells [SharedPool-Worker-1] | 2014-11-08 08:07:14.028000 | 127.0.0.5 |            606
                                              Enqueuing response to /127.0.0.1 [SharedPool-Worker-1] | 2014-11-08 08:07:14.028001 | 127.0.0.5 |           1125
                                                         Message received from /127.0.0.3 [Thread-8] | 2014-11-08 08:07:14.029000 | 127.0.0.1 |           3224
                                                    Sending message to /127.0.0.1 [WRITE-/127.0.0.1] | 2014-11-08 08:07:14.029000 | 127.0.0.5 |           1616
                                           Processing response from /127.0.0.3 [SharedPool-Worker-3] | 2014-11-08 08:07:14.029000 | 127.0.0.1 |           3417
                                                        Message received from /127.0.0.5 [Thread-10] | 2014-11-08 08:07:14.029000 | 127.0.0.1 |           3454
                                                         Message received from /127.0.0.4 [Thread-7] | 2014-11-08 08:07:14.029000 | 127.0.0.1 |           3516
                                           Processing response from /127.0.0.5 [SharedPool-Worker-2] | 2014-11-08 08:07:14.029000 | 127.0.0.1 |           3627
                                           Processing response from /127.0.0.4 [SharedPool-Worker-2] | 2014-11-08 08:07:14.030000 | 127.0.0.1 |           3688
                                                                                    Request complete | 2014-11-08 08:07:14.030347 | 127.0.0.1 |           4347

How consistency affects performance

Changing the consistency level can affect read performance. The tracing output shows that as you change the consistency level from ONE to QUORUM to ALL, performance degrades in from 2585 to 2998 to 5219 microseconds, respectively. If you follow the steps in this tutorial, it is not guaranteed that you will see the same trend because querying a one-row table is a degenerate case, used for example purposes. The difference between QUORUM and ALL is slight in this case, so depending on conditions in the cluster, performance using ALL might be faster than QUORUM.

Under the following conditions, performance using ALL is worse than QUORUM:

  • The data consists of thousands of rows or more.
  • One node is slower than others.
  • A particularly slow node was not selected to be part of the quorum.

Tracing queries on large datasets

You can use probabilistic tracing on databases having at least ten rows, but this capability is intended for tracing through much more data. After configuring probabilistic tracing using the nodetool settraceprobability command, you query the system_traces keyspace.

SELECT * FROM system_traces.events;
Show/hide