Brian Hess

<p>This is the third blog post about dsbulk. &nbsp;The first two blog posts (<a href="/blog/2019/03/datastax-bulk-loader-introduction-and-loading">here</a> and <a href="/blog/2019/04/datastax-bulk-loader-more-loading">here</a>) covered some basic loading examples. &nbsp;This post will delve into some of the common options to load, unloading, and counting.</p>

<hr />
<h2>Example 14: Specifying the connection</h2>

<p>The examples so far have loaded into DSE running on the local machine, which is the default host. &nbsp;We can specify the host IP or name with the -h parameter:</p>

<p><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -h 127.0.0.1</code></p>

<h2>Example 14.1: Specifying the port</h2>

<p>If you are in the unlikely scenario that DSE is listening on a different port than 9042, you can also specify the port using the -port parameter:</p>

<p><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -h 127.0.0.1 -port 9042</code></p>

<h2>Example 14.2: Specifying username/password</h2>

<p>If your cluster is protected with username/password authentication, you can pass those in via the -u and -p parameters:</p>

<p><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -u cassandra -p cassandra</code></p>

<p>NOTE: You should take care when submitting passwords on the command-line. &nbsp;These can show up in system logs, environments, etc.</p>

<h2>Example 14.3: Connecting to Kerberos enabled cluster</h2>

<p>You can use dsbulk also with Kerberos-enabled cluster:</p>

<p><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id --driver.auth.provider DseGSSAPIAuthProvider --driver.auth.saslService --driver.auth.principal -h</code></p>

<p>Where:</p>

<ul>
	<li>service - service name, the same as service value in the ~/.cassandra/cqlshrc;</li>
	<li>host - must be hostname as registered in Kerberos. The same hostname value in the ~/.cassandra/cqlshrc;</li>
	<li>principal - Kerberos principal to use. Could be obtained via klist.</li>
</ul>

<p>See here for more details.</p>

<h2>Example 15: Specifying the Consistency Level</h2>

<p>The default consistency level for dsbulk is LOCAL_ONE. &nbsp;You can specify the desired consistency level via the -cl parameter, which is short for --driver.query.consistency:</p>

<p><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -cl LOCAL_QUORUM </code></p>

<h2>Example 16: Errors and Logging</h2>

<p>Let’s face it, data is never as clean as you think it is. &nbsp;Things happen, errors happen, it’s okay. As we’ve discussed, dsbulk can handle errors by logging them and copying the problematic records to a “bad file” to be looked at later. &nbsp;But, dsbulk has other parameters to control the behavior of errors.</p>

<h2>Example 16.1: Max Errors</h2>

<p>Sometimes the data is so bad that you want to tell dsbulk to stop. &nbsp;You can specify the maximum number of errors with the -maxErrors parameter, which is short for --log.maxErrors, and defaults to 10:</p>

<p><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id --log.maxErrors 3</code></p>

<h2>Example 16.2: Setting the logging directory</h2>

<p>dsbulk logs to a directory named logs in the current directory where dsbulk is invoked. &nbsp;You can specify a different location with the -logDir parameter, which is short for --log.directory:</p>

<p><code><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -logDir /tmp/logs</code></code></p>

<h2>Example 16.3: Setting the logging verbosity</h2>

<p>dsbulk has a way to dial up or down the verbosity of the logging, via --log.verbosity. Valid levels are 0, 1, and 2, from least to most verbose. &nbsp;For example, to get the minimum output:</p>

<p><code><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id --log.verbosity 0</code></code></p>

<h2>Example 17: Monitoring</h2>

<p>dsbulk has a few ways to monitor the progress of dsbulk. &nbsp;The two main ways are reporting to the screen and via JMX.</p>

<h2>Example 17.1: Setting the report rate</h2>

<p>The progress is reported to the screen by default every 5 seconds. &nbsp;This rate can be set with the -reportRate parameter, which is a shortcut for --monitoring.reportRate:</p>

<p><code><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -reportRate 2s</code></code></p>

<p>Or:</p>

<p><code><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -reportRate "2 seconds"</code></code></p>

<h2>Example 17.2: Enabling/Disabling JMX</h2>

<p>By default, dsbulk will provide metrics via JMX. &nbsp;You can disable this via the --monitoring.jmx parameter:</p>

<p><code><code>$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id --monitoring.jmx false</code></code></p>

<hr />
<p>To download the DataStax Bulk Loader click <a href="https://downloads.datastax.com/#bulk-loader" target="_blank">here</a>.</p>

<p>For an intro to unloading, read the next Bulk Loader blog <a href="https://www.datastax.com/blog/2019/06/datastax-bulk-loader-unloading">here</a>.</p>

<p>For basic loading examples, read the previous Bulk Loader blog <a href="https://www.datastax.com/blog/2019/12/datastax-bulk-loader-examples-loading-other-locations">here</a>.</p>

<p><img alt="dsbulk DataStax Enterprise" data-entity-type="file" data-entity-uuid="4488158f-75b7-4c76-bf86-0e0d8406e9fa" src="https://www.datastax.com/sites/default/files/inline-images/dsbulk_0.png" /></p>


DataStax Bulk Loader Pt. 3 — Common Settings

Brian Hess

Discover more

Share

More Technology

Why DataStax Loves LangChain

Getting Started with DataStax Astra DB and Amazon Bedrock

Why You Can't Miss Cassandra Summit

Mitigate the OpenAI Debacle with Open Source and Cloud

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale

Brian Hess

Discover more

Share

More Technology

Why DataStax Loves LangChain

Getting Started with DataStax Astra DB and Amazon Bedrock

Why You Can't Miss Cassandra Summit

Mitigate the OpenAI Debacle with Open Source and Cloud

NoSQL and Vector DBfor Generative AI,Instantly, At Scale

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale