Michaël Figuière

Proper error handling with databases is always a challenge when the safety of your data is involved. Cassandra is no exception to this rule. Thanks to the complete control of consistency, availability and durability offered by Cassandra, error handling turns out to be very flexible and, if done right, will allow you to extend the continuous availability of your cluster even further. But in order to reach this goal, it's important to understand the various kind of errors that can be thrown by the drivers and how to handle them properly.

To remain practical, this article will refer directly to the DataStax Java Driver exceptions and API, but all the concepts explained here can be transposed directly to other DataStax Drivers.

<h2>Error handling in a distributed system</h2>

A typical query in Cassandra can imply message exchanges across more than two machines. This leads to additional error causes compared to a classic client server exchange. Consider the request in the following diagram:

<img alt="Cassandra Request" data-entity-type="file" data-entity-uuid="782be630-843d-4d59-a38d-9ee7706928ef" src="https://www.datastax.com/sites/default/files/inline-images/Cassandra_Request.jpg" />

&nbsp;

In this very typical example of Cassandra request, 4 request-response round trips are performed to complete a query. Errors can happen for any of these steps and the driver provides specific errors to deal with each possible situation:

<code>NoHostAvailableException</code> 
In the situation where the driver is not able to establish a connection to any Cassandra node, either at startup or at a later point in time, the driver will throw a&nbsp;<code>NoHostAvailableException</code>.

<code>UnavailableException</code> 
When a request reaches the coordinator and there is not enough replica alive to achieve the requested consistency level, the driver will throw an&nbsp;<code><a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/exceptions/UnavailableException.html">UnavailableException</a></code>. If you look carefully at this exception, you'll notice that it's possible to get the amount of replicas that were&nbsp;known&nbsp;to be alive when the error was triggered, as well as the amount of replicas that where&nbsp;required&nbsp;by the requested consistency level.

<code>ReadTimeoutException</code> 
Sometimes the coordinator believes that there's enough replicas alive to process a query with the requested consistency level, but for some reasons one or several nodes are too slow to answer within the predefined timeout. This could be due to a temporary overloading of these nodes or even that they've just failed or have been turned off. If the query is a read, a&nbsp;<code>ReadTimeoutException</code>&nbsp;will be thrown. During reads, Cassandra doesn't request data from every replica to minimize internal network traffic. Instead, some replica are only asked for a checksum of the data. A read timeout may occurred even if enough replica have responded to fulfill the consistency level if only checksum responses have been received. In Java,&nbsp;<code><a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/exceptions/ReadTimeoutException.html#wasDataRetrieved()">wasDataRetrieved()</a></code>&nbsp;allows you to check whether you're in this situation or not. Note that by default, the driver will automatically retry the query if only checksum responses have been received, making this cause of read timeout not so likely to reach your code.

<code>WriteTimeoutException</code> 
In a similar situation as above, if the query is a write, then a&nbsp;<code>WriteTimeoutException</code>&nbsp;will be thrown. If the driver differentiates between read and write timeout here, this is because a read is obviously a non mutating operation, whereas a write is likely to be. If a write timeouts at the coordinator level, there is no way to know whether the mutation has been applied or not on the non-answering replica. As we'll see next, the way this error will be handled will thus depends on whether the write operation was idempotent (which is the case of most statements in CQL) or not (for counter updates, and append/prepend updates on lists).

<code>QueryValidationException</code> 
As with any database, when something's wrong in the way a query string has been written, the driver will throw an exception. In the case of the Java Driver, that will be a&nbsp;<code>QueryValidationException</code>.

<h2>When you can't do much</h2>

If the driver throw a&nbsp;<code>QueryValidationException</code>, you obviously can't do anything but fixing your code or checking the compatibility of your query with the Cassandra version you are using here.

In the previous section we've seen how a&nbsp;<code>NoHostAvailableException</code>&nbsp;was raised when the driver isn't able to connect to any node. This is caused by your Cassandra cluster being down or any network connectivity issue between the client and the cluster. When it happens in production it typically means that a manual operation will be required to re-establish the communication between the client and the Cassandra cluster, while ensuring that it's correctly up and running. The driver will automatically reconnect to the Cassandra nodes with regular retries (that you can configure when starting the driver with&nbsp;<code><a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/Cluster.Builder.html#withReconnectionPolicy(com.datastax.driver.core.policies.ReconnectionPolicy)">withReconnectionPolicy()</a></code>).

<h2>Non-idempotent operations</h2>

Most operations are idempotent in Cassandra, but a few aren't: counter updates, append/prepend on list, both because they represent a delta and not an absolute value. When the driver throw a&nbsp;<code>WriteTimeoutException</code>&nbsp;while executing one of these operations, there is no way to know whether the delta has been applied or not. And because the update isn't idempotent retries could cause duplicates.

As a consequence it's a good idea to avoid using these operations to update critical data. For instance CAS should be preferred to counters to update stock quantities in an eCommerce environment, while&nbsp;<code>set</code>&nbsp;should be preferred to&nbsp;<code>list</code>&nbsp;when storing a list of tags. Thus counters are more appropriate for analytics use cases where a slight error margin is acceptable, while list's append/prepend should be used when duplicates can be accepted, otherwise a complete replacement of the list is a safer way to update it.

Additionally, using CQL functions to generate random values such as UUIDs, will make a write query non idempotent:&nbsp;<code>uuid()</code>, for instance, will be evaluated on the server side, and evaluated again if the client makes another try, which will lead to a different value. Therefore, generating UUIDs within your application is generally a better idea. Random value generating functions like&nbsp;<code>uuid()</code>&nbsp;are actually more appropriate in interactive environments such as cqlsh or&nbsp;<a href="http://docs.datastax.com/en/getting_started/doc/developer/devcenter/usingDevCenter.html">DataStax DevCenter</a>.

<h2>CAS operations</h2>

CAS (Compare-And-Set) operations, also called Lightweight Transactions, have been&nbsp;<a href="https://www.datastax.com/dev/blog/lightweight-transactions-in-cassandra-2-0">introduced in Cassandra 2.0</a>. They rely on the&nbsp;<a href="http://en.wikipedia.org/wiki/Paxos_(computer_science)">Paxos protocol</a>&nbsp;to achieve a distributed consensus. These operations have a&nbsp;paxos&nbsp;phase and a&nbsp;commit&nbsp;phase. The consistency level of the former is set on a&nbsp;<code>Statement</code>&nbsp;with&nbsp;<code><a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/Statement.html#setSerialConsistencyLevel(com.datastax.driver.core.ConsistencyLevel)">setSerialConsistencyLevel()</a></code>&nbsp;while the consistency level of the latter is defined with the usual&nbsp;<code><a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/Statement.html#setConsistencyLevel(com.datastax.driver.core.ConsistencyLevel)">setConsistencyLevel()</a></code>&nbsp;method.

If the&nbsp;paxos&nbsp;phase fails, the driver will throw a&nbsp;<code>WriteTimeoutException</code>&nbsp;with a&nbsp;<code>WriteType.CAS</code>&nbsp;as retrieved with&nbsp;<code><a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/exceptions/WriteTimeoutException.html#getWriteType()">WriteTimeoutException#getWriteType()</a></code>. In this situation you can't know if the CAS operation has been applied so you need to retry it in order to fallback on a stable state. Because lightweight transactions are much more expensive that regular updates, the driver doesn't automatically retry it for you. The&nbsp;paxos&nbsp;phase can also lead to an&nbsp;<code>UnavailableException</code>&nbsp;if not enough replicas are available. In this situation, retries won't help as only&nbsp;<code>SERIAL</code>&nbsp;and&nbsp;<code>LOCAL_SERIAL</code>&nbsp;consistencies are available.

The&nbsp;commit&nbsp;phase is then similar to regular Cassandra writes in the sense that it will throw an&nbsp;<code>UnavailableException</code>&nbsp;or a&nbsp;<code>WriteTimeoutException</code>&nbsp;if the amount of required replicas or acknowledges isn't met. In this situation rather than retrying the entire CAS operation, you can simply ignore this error if you make sure to use&nbsp;<code>setConsistencyLevel(ConsistencyLevel.SERIAL)</code>&nbsp;on the subsequent read statements on the column that was touched by this transaction, as it will force Cassandra to commit any remaining uncommitted Paxos state before proceeding with the read. That being said, it probably won't be easy to organize an application to use&nbsp;<code>SERIAL</code>&nbsp;reads after a CAS write failure, so you may prefer another alternative such as an entire retry of the CAS operation.

<h2>Batch queries</h2>

Cassandra 1.2 introduced&nbsp;<a href="https://www.datastax.com/dev/blog/atomic-batches-in-cassandra-1-2">atomic batches</a>, which rely on a batch log to guarantee that all the mutations in the batch will eventually be applied. This atomicity is useful as it allows developers to execute several writes on different partitions without having to worry on which part will be applied and which won't: either all or none of the mutations will be eventually written.

If a timeout occurs when a batch is executed, the developer has different options depending on the type of write that timed out (see&nbsp;<code><a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/exceptions/WriteTimeoutException.html#getWriteType()">WriteTimeoutException#getWriteType()</a></code>):

<ul>
	<li><code>BATCH_LOG</code>: a timeout occurred while the coordinator was waiting for the batch log replicas to acknowledge the log. Thus the batch may or may not be applied. By default, the driver will retry the batch query once, when it's notified that such a timeout occurred. So if you receive this error, you may want to retry again, but that's already a bad smell that the coordinator has been unlucky at picking replicas twice.</li>
	<li><code>BATCH</code>: a timeout occurred while reaching replicas for one of the changes in an atomic batch, after an entry has been successfully written to the batch log. Cassandra will thus ensure that this batch will get eventually written to the appropriate replicas and the developer doesn't have to do anything. Note however that this error still means that all the columns haven't all been updated&nbsp;yet. So if the immediate consistency of these writes is required in the business logic to be executed, you probably want to consider an alternate end, or a warning message to the end user.</li>
	<li><code>UNLOGGED_BATCH</code>: the coordinator met a timeout while reaching the replicas for a write query being part of an unlogged batch. This batch isn't guaranteed to be atomic as no batch log entry is written, thus the parts of the batch that will or will not be applied are unknown. A retry of the entire batch will be required to fall back on a known state.</li>
</ul>

<h2>When to retry</h2>

Retries can be used in Cassandra thanks to the fact that most queries are idempotent. If the driver sends a write query again&nbsp;with the same the same timestamp, it won't overwrite another update that may have arrived in between, as the last timestamp will always win.

While retries can be expensive in some situation and should be considered with care, it's still a useful error handling strategy in many cases. In general, retries can be useful when receiving a timeout exception or an&nbsp;<code>UnavailableException</code>&nbsp;stating that the amount of received acknowledges or available replicas is greater or equal than one and that you're willing to downgrade the consistency level of a query, and to loose the consistency guarantee that came with it.

DataStax Drivers come with a way to automatically handle retries in your application:&nbsp;<code><a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/policies/RetryPolicy.html">RetryPolicy</a></code>. This simple interface lets you implement the logic that you want to apply to any given statement (note that you can&nbsp;<a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/Statement.html#setRetryPolicy(com.datastax.driver.core.policies.RetryPolicy)">set a particular&nbsp;<code>RetryPolicy</code></a>&nbsp;to a statement). When handling retries with this interface, timestamps will be set properly to ensure that queries are idempotent.

By default, the driver uses an implementation of this interface, the&nbsp;<code><a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/policies/DefaultRetryPolicy.html">DefaultRetryPolicy</a></code>, that only performs retries in two cases that we consider safe and almost always efficient:

<ul>
	<li>On a read timeout, if enough replica replied but data was not retrieved.</li>
	<li>On a write timeout, if a timeout occurs while writing the distributed log used by batch statements.</li>
</ul>

A&nbsp;<code>RetryPolicy</code>&nbsp;is provided to handle consistency level downgrading in a straightforward way:&nbsp;<code><a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/policies/DowngradingConsistencyRetryPolicy.html">DowngradingConsistencyRetryPolicy</a></code>. This policy will retry failed queries with the best consistency level that is likely to work considering the received acknowledges or available replicas. Note that if you use this policy, the retry will be handled transparently with a lowered consistency so the developer&nbsp;must&nbsp;check the&nbsp;<a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/ExecutionInfo.html#getAchievedConsistencyLevel()">consistency level that was achieved</a>&nbsp;to probably display a warning to the user or any other such kind of downgrading experience strategy. You may also want to consider the&nbsp;<code><a href="https://docs.datastax.com/en/drivers/java/2.1/com/datastax/driver/core/policies/LoggingRetryPolicy.html">LoggingRetryPolicy</a></code>&nbsp;to automatically log the retry. Finally, note that&nbsp;if you find yourself using a downgraded consistency level and feeling OK with it, then you should probably wonder whether the original consistency level wasn't too excessive for your actual requirement.

To conclude this article, let's summarize these few takeaway advices:

<ul>
	<li>Understand the difference between&nbsp;<code>NoHostAvailableException</code>,&nbsp;<code>UnavailableException</code>,&nbsp;<code>Read/WriteTimeoutException</code>.</li>
	<li>Write timeout of non-idempotent operations can lead to unknown states, don't use these operation for critical data.</li>
	<li>Consider using a consistency level&nbsp;<code>SERIAL</code>&nbsp;for reads on a column updated by a CAS operation, if the extra cost on reads is acceptable, to cope with&nbsp;commit&nbsp;phase errors in a simple way.</li>
	<li>Atomic batches come with a very handy guarantee that updates will eventually be applied, be sure not to reinvent this mechanism yourself on the client side.</li>
	<li>Use retries with care, in a sparse manner.</li>
</ul>

Cassandra error handling done right

Michaël FiguièreSoftware Engineer

Discover more

Share

More Technology

Getting Started with DataStax Astra DB and Amazon Bedrock

Why You Can't Miss Cassandra Summit

Mitigate the OpenAI Debacle with Open Source and Cloud

Supercharge AI Assistants with Superagent and DataStax

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale