Brian Hess

<p>This is the third blog post about dsbulk. &nbsp;The first two blog posts (<a href="/blog/2019/03/datastax-bulk-loader-introduction-and-loading">here</a> and <a href="/blog/2019/04/datastax-bulk-loader-more-loading">here</a>) covered some basic loading examples. &nbsp;This post will delve into some of the common options to load, unloading, and counting.</p>

<hr />
<h2>Example 14: Specifying the connection</h2>

<p>The examples so far have loaded into DSE running on the local machine, which is the default host. &nbsp;We can specify the host IP or name with the -h parameter:</p>

<p><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -h 127.0.0.1</code></p>

<h2>Example 14.1: Specifying the port</h2>

<p>If you are in the unlikely scenario that DSE is listening on a different port than 9042, you can also specify the port using the -port parameter:</p>

<p><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -h 127.0.0.1 -port 9042</code></p>

<h2>Example 14.2: Specifying username/password</h2>

<p>If your cluster is protected with username/password authentication, you can pass those in via the -u and -p parameters:</p>

<p><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -u cassandra -p cassandra</code></p>

<p>NOTE: You should take care when submitting passwords on the command-line. &nbsp;These can show up in system logs, environments, etc.</p>

<h2>Example 14.3: Connecting to Kerberos enabled cluster</h2>

<p>You can use dsbulk also with Kerberos-enabled cluster:</p>

<p><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id --driver.auth.provider DseGSSAPIAuthProvider --driver.auth.saslService --driver.auth.principal -h</code></p>

<p>Where:</p>

<ul>
	<li>service - service name, the same as service value in the ~/.cassandra/cqlshrc;</li>
	<li>host - must be hostname as registered in Kerberos. The same hostname value in the ~/.cassandra/cqlshrc;</li>
	<li>principal - Kerberos principal to use. Could be obtained via klist.</li>
</ul>

<p>See here for more details.</p>

<h2>Example 15: Specifying the Consistency Level</h2>

<p>The default consistency level for dsbulk is LOCAL_ONE. &nbsp;You can specify the desired consistency level via the -cl parameter, which is short for --driver.query.consistency:</p>

<p><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -cl LOCAL_QUORUM </code></p>

<h2>Example 16: Errors and Logging</h2>

<p>Let’s face it, data is never as clean as you think it is. &nbsp;Things happen, errors happen, it’s okay. As we’ve discussed, dsbulk can handle errors by logging them and copying the problematic records to a “bad file” to be looked at later. &nbsp;But, dsbulk has other parameters to control the behavior of errors.</p>

<h2>Example 16.1: Max Errors</h2>

<p>Sometimes the data is so bad that you want to tell dsbulk to stop. &nbsp;You can specify the maximum number of errors with the -maxErrors parameter, which is short for --log.maxErrors, and defaults to 10:</p>

<p><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id --log.maxErrors 3</code></p>

<h2>Example 16.2: Setting the logging directory</h2>

<p>dsbulk logs to a directory named logs in the current directory where dsbulk is invoked. &nbsp;You can specify a different location with the -logDir parameter, which is short for --log.directory:</p>

<p><code><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -logDir /tmp/logs</code></code></p>

<h2>Example 16.3: Setting the logging verbosity</h2>

<p>dsbulk has a way to dial up or down the verbosity of the logging, via --log.verbosity. Valid levels are 0, 1, and 2, from least to most verbose. &nbsp;For example, to get the minimum output:</p>

<p><code><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id --log.verbosity 0</code></code></p>

<h2>Example 17: Monitoring</h2>

<p>dsbulk has a few ways to monitor the progress of dsbulk. &nbsp;The two main ways are reporting to the screen and via JMX.</p>

<h2>Example 17.1: Setting the report rate</h2>

<p>The progress is reported to the screen by default every 5 seconds. &nbsp;This rate can be set with the -reportRate parameter, which is a shortcut for --monitoring.reportRate:</p>

<p><code><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -reportRate 2s</code></code></p>

<p>Or:</p>

<p><code><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -reportRate "2 seconds"</code></code></p>

<h2>Example 17.2: Enabling/Disabling JMX</h2>

<p>By default, dsbulk will provide metrics via JMX. &nbsp;You can disable this via the --monitoring.jmx parameter:</p>

<p><code><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id --monitoring.jmx false</code></code></p>

<hr />
<p>To download the DataStax Bulk Loader click <a href="https://downloads.datastax.com/#bulk-loader" target="_blank">here</a>.</p>

<p>For an intro to unloading, read the next Bulk Loader blog <a href="https://www.datastax.com/blog/2019/06/datastax-bulk-loader-unloading">here</a>.</p>

<p>For basic loading examples, read the previous Bulk Loader blog <a href="https://www.datastax.com/blog/2019/12/datastax-bulk-loader-examples-loading-other-locations">here</a>.</p>

<p><img alt="dsbulk DataStax Enterprise" data-entity-type="file" data-entity-uuid="4488158f-75b7-4c76-bf86-0e0d8406e9fa" src="https://www.datastax.com/sites/default/files/inline-images/dsbulk_0.png" /></p>


ViewingTechnical How To’s

DataStax Bulk Loader Pt. 3 — Common Settings

Sign up for our Developer Newsletter