Sam Tunnicliffe

<p>Following on from recent posts on testing in the Apache Cassandra project with dtests and&nbsp;<a href="https://www.datastax.com/dev/blog/testing-apache-cassandra-with-jepsen" target="_blank">Jepsen</a>, I wanted to look at an interesting tool which we've recently begun to explore in our unit tests.</p>

<p>There are some things which are notoriously difficult to cover in unit testing. Verifying behaviors without easily observable side effects is one such case, for example verifying that a particular code path is followed under specific conditions. This type of observability can be increased by refactoring and employing techniques such as dependency injection, but this often comes at the expense of clarity and concision in the code.</p>

<p>Another example is fault injection testing, where pathological conditions are artificially induced at test runtime. This can be extremely useful to exercise those corners of a codebase which deal with error handling, particularly when those errors are difficult to reproduce in a test environment. Verifying correct responses to scenarios such as a disk filling up or a network partition are clearly crucial to developing robust systems, yet these are often hard to model in unit tests. Higher level testing frameworks can provide mechanisms for creating or simulating these scenarios, such as&nbsp;<a href="https://github.com/aphyr/jepsen/blob/master/doc/nemesis.md" target="_blank">Jepsen's nemeses</a>, but as with the observability problem, unit tests have often had to rely on dependency injection and mocks or stubs to force execution of error handling code paths.</p>

<p><a href="http://byteman.jboss.org/" target="_blank">Byteman</a>&nbsp;is an&nbsp;<a href="https://github.com/bytemanproject/byteman" target="_blank">open source</a>&nbsp;tool, primarily developed by&nbsp;<a href="http://www.jboss.org/" target="_blank">JBoss</a>, which enables additional Java code to be injected into a running JVM. From the project's home page:&nbsp;<em>"You can inject code almost anywhere you want and there is no need to prepare the original source code in advance. You can even remove injected code and reinstall different changes while the application continues to execute."</em></p>

<p>Injections are known as rules and scripted using a simple DSL with primitives for tracing and modifying as well as for defining trigger points and conditions in the existing code. This clearly meshes well with both the observability and fault injection concepts and in fact Byteman ships with a JUnit test runner to support integration with the test fixtures through annotations.</p>

<p>Let's look at a recently committed Cassandra unit test which uses Byteman.The intent in the&nbsp;<a href="https://github.com/apache/cassandra/blob/037d24efdf83bd2736556f9880c5e1f6be48fa77/test/unit/org/apache/cassandra/hints/HintsBufferPoolTest.java" target="_blank">test case</a>&nbsp;added for&nbsp;<a href="https://issues.apache.org/jira/browse/CASSANDRA-10972" target="_blank">CASSANDRA-10972</a>&nbsp;is to assert that the&nbsp;<code>HintsBufferPool</code>&nbsp;provides some backpressure to its callers by drawing its write buffers from a&nbsp;<code>BlockingQueue</code>.</p>

<p>The annotation on the test method specifies an action and defines the point at which to execute it:</p>

<table border="0" cellpadding="0" cellspacing="0">
	<tbody>
		<tr>
			<td>
			<p>1</p>

			<p>2</p>

			<p>3</p>

			<p>4</p>

			<p>5</p>

			<p>6</p>

			<p>7</p>
			</td>
			<td>
			<p><code>@Test</code></p>

			<p><code>@BMRule</code><code>(name = </code><code>"Greatest name in the world"</code><code>,</code></p>

			<p><code>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code><code>targetClass=</code><code>"HintsBufferPool"</code><code>,</code></p>

			<p><code>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code><code>targetMethod=</code><code>"switchCurrentBuffer"</code><code>,</code></p>

			<p><code>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code><code>targetLocation=</code><code>"AT INVOKE java.util.concurrent.BlockingQueue.take"</code><code>,</code></p>

			<p><code>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</code><code>action=</code><code>"org.apache.cassandra.hints.HintsBufferPoolTest.blockedOnBackpressure = true;"</code><code>)</code></p>

			<p><code>public</code> <code>void</code> <code>testBackpressure() </code><code>throws</code> <code>Exception</code></p>
			</td>
		</tr>
	</tbody>
</table>

<p>We specify an action to perform when our rule is triggered:</p>

<table border="0" cellpadding="0" cellspacing="0">
	<tbody>
		<tr>
			<td>
			<p>6</p>
			</td>
			<td>
			<p><code>action="org.apache.cassandra.hints.HintsBufferPoolTest.blockedOnBackpressure = </code><code>true</code></p>
			</td>
		</tr>
	</tbody>
</table>

<p>which simply flips a boolean flag in the test case. Next, the trigger point:</p>

<table border="0" cellpadding="0" cellspacing="0">
	<tbody>
		<tr>
			<td>
			<p>3</p>

			<p>4</p>

			<p>5</p>
			</td>
			<td>
			<p><code>targetClass=</code><code>"HintsBufferPool"</code><code>,</code></p>

			<p><code>targetMethod=</code><code>"switchCurrentBuffer"</code><code>,</code></p>

			<p><code>targetLocation=</code><code>"AT INVOKE java.util.concurrent.BlockingQueue.take"</code><code>,</code></p>
			</td>
		</tr>
	</tbody>
</table>

<p>These attributes represent the code coordinates at which to perform the action; in this case, during a call&nbsp;<code>HintsBufferPool::switchCurrentBuffer</code>. More specifically, during execution of that method whenever&nbsp;<code>BlockingQueue::take</code>&nbsp;is called, the defined action is performed. Note that&nbsp;<code>switchCurrentBuffer</code>&nbsp;is a private method, Byteman can inspect and inject absolutely anywhere in application, library or even Java runtime code. Several options are available when specifying the&nbsp;<code>targetLocation</code>, including directly before or after execution of the target method, when named variables are read or written, when particular method calls are made (as in this example) and even at specific lines in the source.</p>

<p>Finally, the test asserts that the flag was set, indicating that at some point during&nbsp;<code>switchCurrentBuffer</code>&nbsp;the pool did draw from the recycled buffer queue:</p>

<table border="0" cellpadding="0" cellspacing="0">
	<tbody>
		<tr>
			<td>
			<p><code>assertTrue(blockedOnBackpressure);</code></p>
			</td>
		</tr>
	</tbody>
</table>

<p>In future, expect to see more fault injection in Cassandra's unit tests and probably also in dtests. Hooks are already in place in&nbsp;<a href="https://www.datastax.com/dev/blog/ccm-a-development-tool-for-creating-local-cassandra-clusters" target="_blank">CCM</a>&nbsp;(the library used by dtests to manage local clusters) for starting nodes with the Byteman agent installed and submitting scripts to those nodes. Using them to deterministically invoke corner cases in a dtest cluster will help expand our test coverage and complement the Jepsen tests.</p>

<p>Further reading:</p>

<ul>
	<li><a href="http://downloads.jboss.org/byteman/3.0.2/ProgrammersGuide.html">http://downloads.jboss.org/byteman/3.0.2/ProgrammersGuide.html</a></li>
	<li><a href="https://developer.jboss.org/wiki/FaultInjectionTestingWithByteman">https://developer.jboss.org/wiki/FaultInjectionTestingWithByteman</a></li>
	<li><a href="https://developer.jboss.org/wiki/BMUnitUsingBytemanWithJUnitOrTestNGFromMavenAndAnt">https://developer.jboss.org/wiki/BMUnitUsingBytemanWithJUnitOrTestNGFromMavenAndAnt</a></li>
	<li><a href="http://blog.eisele.net/2015/02/byteman-swiss-army-knife-for-byte-code.html">http://blog.eisele.net/2015/02/byteman-swiss-army-knife-for-byte-code.html</a></li>
</ul>


Cassandra Unit Testing with Byteman

Sam Tunnicliffe

Discover more

Share

More Technology

Why DataStax Loves LangChain

Getting Started with DataStax Astra DB and Amazon Bedrock

Why You Can't Miss Cassandra Summit

Mitigate the OpenAI Debacle with Open Source and Cloud

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale

Sam Tunnicliffe

Discover more

Share

More Technology

Why DataStax Loves LangChain

Getting Started with DataStax Astra DB and Amazon Bedrock

Why You Can't Miss Cassandra Summit

Mitigate the OpenAI Debacle with Open Source and Cloud

NoSQL and Vector DBfor Generative AI,Instantly, At Scale

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale