Cassandra Unit Testing with Byteman

Following on from recent posts on testing in the Apache Cassandra project with dtests and Jepsen, I wanted to look at an interesting tool which we've recently begun to explore in our unit tests.

There are some things which are notoriously difficult to cover in unit testing. Verifying behaviors without easily observable side effects is one such case, for example verifying that a particular code path is followed under specific conditions. This type of observability can be increased by refactoring and employing techniques such as dependency injection, but this often comes at the expense of clarity and concision in the code.

Another example is fault injection testing, where pathological conditions are artificially induced at test runtime. This can be extremely useful to exercise those corners of a codebase which deal with error handling, particularly when those errors are difficult to reproduce in a test environment. Verifying correct responses to scenarios such as a disk filling up or a network partition are clearly crucial to developing robust systems, yet these are often hard to model in unit tests. Higher level testing frameworks can provide mechanisms for creating or simulating these scenarios, such as Jepsen's nemeses, but as with the observability problem, unit tests have often had to rely on dependency injection and mocks or stubs to force execution of error handling code paths.

Byteman is an open source tool, primarily developed by JBoss, which enables additional Java code to be injected into a running JVM. From the project's home page: "You can inject code almost anywhere you want and there is no need to prepare the original source code in advance. You can even remove injected code and reinstall different changes while the application continues to execute."

Injections are known as rules and scripted using a simple DSL with primitives for tracing and modifying as well as for defining trigger points and conditions in the existing code. This clearly meshes well with both the observability and fault injection concepts and in fact Byteman ships with a JUnit test runner to support integration with the test fixtures through annotations.

Let's look at a recently committed Cassandra unit test which uses Byteman.The intent in the test case added for CASSANDRA-10972 is to assert that the HintsBufferPool provides some backpressure to its callers by drawing its write buffers from a BlockingQueue.

The annotation on the test method specifies an action and defines the point at which to execute it:

@Test

@BMRule(name = "Greatest name in the world",

        targetClass="HintsBufferPool",

        targetMethod="switchCurrentBuffer",

        targetLocation="AT INVOKE java.util.concurrent.BlockingQueue.take",

        action="org.apache.cassandra.hints.HintsBufferPoolTest.blockedOnBackpressure = true;")

public void testBackpressure() throws Exception

We specify an action to perform when our rule is triggered:

action="org.apache.cassandra.hints.HintsBufferPoolTest.blockedOnBackpressure = true

which simply flips a boolean flag in the test case. Next, the trigger point:

targetClass="HintsBufferPool",

targetMethod="switchCurrentBuffer",

targetLocation="AT INVOKE java.util.concurrent.BlockingQueue.take",

These attributes represent the code coordinates at which to perform the action; in this case, during a call HintsBufferPool::switchCurrentBuffer. More specifically, during execution of that method whenever BlockingQueue::take is called, the defined action is performed. Note that switchCurrentBuffer is a private method, Byteman can inspect and inject absolutely anywhere in application, library or even Java runtime code. Several options are available when specifying the targetLocation, including directly before or after execution of the target method, when named variables are read or written, when particular method calls are made (as in this example) and even at specific lines in the source.

Finally, the test asserts that the flag was set, indicating that at some point during switchCurrentBuffer the pool did draw from the recycled buffer queue:

assertTrue(blockedOnBackpressure);

In future, expect to see more fault injection in Cassandra's unit tests and probably also in dtests. Hooks are already in place in CCM (the library used by dtests to manage local clusters) for starting nodes with the Byteman agent installed and submitting scripts to those nodes. Using them to deterministically invoke corner cases in a dtest cluster will help expand our test coverage and complement the Jepsen tests.

Further reading:

• http://downloads.jboss.org/byteman/3.0.2/ProgrammersGuide.html
• https://developer.jboss.org/wiki/FaultInjectionTestingWithByteman
• https://developer.jboss.org/wiki/BMUnitUsingBytemanWithJUnitOrTestNGFromMavenAndAnt
• http://blog.eisele.net/2015/02/byteman-swiss-army-knife-for-byte-code.html