Consistency and availability: having your cake and eating it, with Cassandra
Today’s applications need to look beyond relational
Modern applications are far more demanding than their grandparents in the 90s. Whether it’s Netflix delivering shows to you every night, or Outbrain providing contextual story placement every time you read a news article online, or OpenWave delivering five nines for your email and SMS, modern business is always on, and downtime is unacceptable.
Yesterday’s databases are not adequate for modern applications because their architects in the early 1980s could not imagine the scale at which we operate today. Their design choices worked well for nearly two decades, but now make it impossible for their products to adapt to today’s demands. Specifically, relational databases are designed around the concept of “ACID” transactions, where changes made by one user are rigorously prohibited from interfering with any other. This is called a strictly consistent system, and computer scientists have proved in the CAP theorem (Consistency, Availability, Partition tolerance) that a system cannot offer at the same time both perfect information and uninterruptible availability.
It’s easy to see why this is intuitively: the way to deliver perfect consistency is to assign each piece of data to a “master”, in charge of ensuring only one user can change it at a time. When any such master has a network, power, or any other kind of failure, the data it is responsible for becomes inaccessible until a new master is selected. Whether failover takes seconds or minutes or even hours, downtime costs you money and tarnishes your brand.
But even when everything works correctly, perfect consistency poses problems for modern applications. Recall that Google’s research shows delays as small as 100s of milliseconds measurably reduce user engagement; at the speed of light, it takes about 120 milliseconds to travel from New York to London and back. Global applications require geo-located data across multiple datacenters, which is good policy in case of disaster (as when Outbrain weathered Hurricane Sandy), and is increasingly a basic feature of responsive application design. These datacenters need to be able to operate independently to achieve the necessary performance.
Designing for availability
What is the alternative? Instead of a single master meting out access to each piece of data, you can replicate your data to multiple machines, each of which is authorized to answer questions about and make changes to it. If any of these equal peers goes down, the others can carry on with no interruption, ensuring availability with no failover period. The downside is that these peers coordinate with each other after making changes: if they try to coordinate before, then we’ve actually introduced a new kind of master, with more moving parts but the same problems as the single master case. Thus, it is possible for an application to act on out of date information and compensate for it later, if necessary.
Let’s look at an example. An online retailer would prefer to sell exactly as many widgets as it has in inventory. If there is one widget left, and two customers want to buy it at exactly the same time, it wants to pick one to “win” the widget, and to tell the other that it’s out of stock. This is possible with a consistent database, but only if we’re also willing to accept that nobody will be able to buy anything when the master is down. Is that a good trade? Amazon says, “customers should be able to view and add items to their shopping cart even if disks are failing, network routes are flapping, or data centers are being destroyed by tornadoes.1” So availability is more important than consistency; if two customers are occasionally both permitted to buy the last item, they can compensate by sending an email acknowledging the error and canceling the order.
Banking is another example often thought to require strict consistency. Surely my checking account should never be negative! But in fact, modern ATMs are designed to tolerate network outages and reconcile any changes made while offline. If this results in a negative balance, or overdraft, appropriate action can be taken with the customer.
The best of both worlds
The general pattern, then, is that it is better for modern applications to optimize for availability. That is the problem we designed Cassandra to solve. But exceptions do exist. One example may be user account creation: the options for later disentangling two users assigned the same account are unsatisfactory, so it is better to take the performance penalty and availability risk to ensure we never have to worry about that situation.2 For the first few years of Cassandra’s existence, it was not uncommon to see Cassandra applications with a secondary MySQL or Redis installation nearby for the corners of the application requiring that supplement.
But starting with Cassandra 2.0 (and DataStax Enterprise 4.0), Cassandra has offered Lightweight Transactions to allow opting in to a strictly consistent paradigm as needed. This delivers cost savings, improved reliability, and greater productivity to applications that need to live in both worlds. The key insight is that while Cassandra cannot offer both desired quantities simultaneously, it can offer each at different times, as required by the application.
Six months later, we are starting to see the power of this design gain mainstream momentum. Twitter independently arrived at a similar conclusion while working on Manhattan, announced today. Other NoSQL players will likely follow suit, but Cassandra and DataStax Enterprise offer the best of both consistency and availability today, in production-ready form.
2 Technical readers may note that we could also assign users a surrogate key such as a UUID to avoid natural key collisions.