Alexandre Dutra

<p>Last year, we made our bulk loading utility and Apache Kafka™ connector available for free use with Cassandra and now these tools are open source and Apache-licensed. You can find and browse the source for these tools at the GitHub locations below.</p>

<ul>
	<li><a href="https://github.com/datastax/dsbulk">https://github.com/datastax/dsbulk</a></li>
	<li><a href="https://github.com/datastax/kafka-sink">https://github.com/datastax/kafka-sink</a></li>
</ul>

<p>Feel free to check out the code, fork the repositories or build the tools from the sources. You can also use GitHub issues to request new features or report any bugs; and of course, if you would like to contribute, pull-requests are most welcome. You can also find us at <a href="https://community.datastax.com">community.datastax.com</a> for any general questions, see you there!</p>

<h2>DSBulk at a glance</h2>

<p>In the latest DSBulk version 1.6.0, you'll find a few nice improvements, bug fixes and sensibly better throughput for most unload operations. See the <a href="https://docs.datastax.com/en/dsbulk/doc/dsbulk/releaseNotes/RelNotesdsbulk.html#RelNotesdsbulk">release notes</a> and <a href="https://docs.datastax.com/en/dsbulk/doc/index.html">documentation</a> for details.</p>

<p>The libraries that power this tool are also now available from Maven Central under the <a href="https://search.maven.org/search?q=g:com.datastax.oss%20dsbulk">group ID com.datastax.oss</a>. This means that it is now possible to invoke loading, unloading, and counting operations programmatically in your Java applications!</p>

<p>Let's see a quick example. Assume that we want to programmatically invoke a "load" operation to ingest many CSV files into Cassandra. First, we need to include a few dependencies:</p>

<div class="code">
<div class="code__heading"><span>Bash</span><button class="code__copy">Copy</button></div>

<div class="code__body">
<pre>
<code class="lang-xml">&lt;!- The Runner; should always be included --&gt;
&lt;dependency&gt;
    &lt;groupId&gt;com.datastax.oss&lt;/groupId&gt;
    &lt;artifactId&gt;dsbulk-runner&lt;/artifactId&gt;
    &lt;version&gt;1.6.0&lt;/version&gt;
&lt;/dependency&gt;
&lt;!- The Load Workflow: the operation we want to run --&gt;
&lt;dependency&gt;
    &lt;groupId&gt;com.datastax.oss&lt;/groupId&gt;
    &lt;artifactId&gt;dsbulk-workflow-load&lt;/artifactId&gt;
    &lt;version&gt;1.6.0&lt;/version&gt;
&lt;/dependency&gt;
&lt;!- The Connector: CSV in our case --&gt;
&lt;dependency&gt;
    &lt;groupId&gt;com.datastax.oss&lt;/groupId&gt;
    &lt;artifactId&gt;dsbulk-connectors-csv&lt;/artifactId&gt;
    &lt;version&gt;1.6.0&lt;/version&gt;
&lt;/dependency&gt;
&lt;!-- Two components required by the Load Workflow: an Executor and a Batcher --&gt;
&lt;dependency&gt;
    &lt;groupId&gt;com.datastax.oss&lt;/groupId&gt;
    &lt;artifactId&gt;dsbulk-executor-reactor&lt;/artifactId&gt;
    &lt;version&gt;1.6.0&lt;/version&gt;
&lt;/dependency&gt;
&lt;dependency&gt;
    &lt;groupId&gt;com.datastax.oss&lt;/groupId&gt;
    &lt;artifactId&gt;dsbulk-batcher-reactor&lt;/artifactId&gt;
    &lt;version&gt;1.6.0&lt;/version&gt;
&lt;/dependency&gt;</code></pre>
</div>
</div>

<p>With the above dependencies included in your application, invoking DSBulk becomes a piece of cake:</p>

<div class="code">
<div class="code__heading"><span>Bash</span><button class="code__copy">Copy</button></div>

<div class="code__body">
<pre>
<code class="lang-java">DataStaxBulkLoader dsbulk = new DataStaxBulkLoader(
  "load",
  "-h"  , "host1.com",         // contact point
  "-k"  , "my_keyspace",       // target keyspace
  "-t"  , "my_table",          // target table
  "-url", "/path/to/my/files"  // CSV sources
);
ExitStatus status = dsbulk.run();
System.out.println(status);</code></pre>
</div>
</div>

<p>And voilà! The above triggers the load operation and prints the final exit status. All the benefits from the bulk loader's reactive, highly-efficient threading model are now available in your application with very little code. Log files and other artifacts generated by the tool during the operation would be located in the usual locations on your filesystem. (Please note though: the programmatic use of the bulk loader is new and considered in beta status for now)</p>

<p>However if you are planning to use the bulk loader as a standalone, ready-to-use tool, instead of downloading individual jars from Maven Central, we recommend using DSBulk's distribution bundle from <a href="https://downloads.datastax.com/#bulk-loader">our download server</a>; the distribution bundle contains everything that is required for the tool to be executed from the command line.</p>

<h2>Kafka Sink Connector at a glance</h2>

<p>Like the bulk loader, the Kafka Sink Connector 1.4.0 ships with a few improvements, see the <a href="https://docs.datastax.com/en/dsbulk/doc/dsbulk/releaseNotes/RelNotesdsbulk.html#RelNotesdsbulk">release notes</a> for details or browse the <a href="https://docs.datastax.com/en/kafka/doc/">documentation</a>.</p>

<p>In the process of making our Kafka Sink Connector open-source, we had to change a few things. For end users, the most visible change is the connector class. If your configuration file had the below entry:</p>

<p><code>connector.class=com.datastax.kafkaconnector.DseSinkConnector</code></p>

<p>You should now change it to:</p>

<p><code>connector.class=com.datastax.oss.kafka.sink.CassandraSinkConnector</code></p>

<p>But don't worry! The old connector class still works, but the connector will print a deprecation warning to the logs. Apart from that, your existing configuration should just work the same way.</p>

<p>One noteworthy improvement is the ability to specify which errors you want to ignore when an error occurs during the processing of a Kafka record. You now have 3 possible values for the <code>ignoreErrors</code> option:</p>

<ol>
	<li><code>NONE</code>: No errors are ignored, every error would make the connector stop; use this option if you are sure that your Kafka records are fully compliant with the database table schema, and that your database is always available.</li>
	<li><code>DRIVER</code>: Only driver errors are ignored and logged (that is, Kafka records that could not be written to the database); other errors (for example, mapping errors) would make the connector stop. Use this option if you know that your Kafka records should be compliant with the schema, but don't want the connector to stop if the database is unreachable or is experiencing trouble servicing requests.</li>
	<li><code>ALL</code>: All errors are ignored and logged; the connector would never stop, neither because of an invalid Kafka record, nor because the database is unreachable.</li>
</ol>

<p>You can download the Kafka Sink Connector for Cassandra from <a href="https://downloads.datastax.com/#akc">our download server</a> or from <a href="https://www.confluent.io/hub/datastax/kafka-connect-dse">Confluent Hub</a>.</p>

<p>Enjoy!</p>


Open Source FTW: New Tools For Apache Cassandra™

Alexandre Dutra

Discover more

Share

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale

Alexandre Dutra

Discover more

Share

NoSQL and Vector DBfor Generative AI,Instantly, At Scale

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale