DataStax Bulk Loader Pt. 3 — Common Settings
This is the third blog post about dsbulk. The first two blog posts (here and here) covered some basic loading examples. This post will delve into some of the common options to load, unloading, and counting.
Example 14: Specifying the connection
The examples so far have loaded into DSE running on the local machine, which is the default host. We can specify the host IP or name with the -h parameter:
$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -h 127.0.0.1
Example 14.1: Specifying the port
If you are in the unlikely scenario that DSE is listening on a different port than 9042, you can also specify the port using the -port parameter:
$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -h 127.0.0.1 -port 9042
Example 14.2: Specifying username/password
If your cluster is protected with username/password authentication, you can pass those in via the -u and -p parameters:
$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -u cassandra -p cassandra
NOTE: You should take care when submitting passwords on the command-line. These can show up in system logs, environments, etc.
Example 14.3: Connecting to Kerberos enabled cluster
You can use dsbulk also with Kerberos-enabled cluster:
$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id --driver.auth.provider DseGSSAPIAuthProvider --driver.auth.saslService --driver.auth.principal -h
Where:
- service - service name, the same as service value in the ~/.cassandra/cqlshrc;
- host - must be hostname as registered in Kerberos. The same hostname value in the ~/.cassandra/cqlshrc;
- principal - Kerberos principal to use. Could be obtained via klist.
See here for more details.
Example 15: Specifying the Consistency Level
The default consistency level for dsbulk is LOCAL_ONE. You can specify the desired consistency level via the -cl parameter, which is short for --driver.query.consistency:
$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -cl LOCAL_QUORUM
Example 16: Errors and Logging
Let’s face it, data is never as clean as you think it is. Things happen, errors happen, it’s okay. As we’ve discussed, dsbulk can handle errors by logging them and copying the problematic records to a “bad file” to be looked at later. But, dsbulk has other parameters to control the behavior of errors.
Example 16.1: Max Errors
Sometimes the data is so bad that you want to tell dsbulk to stop. You can specify the maximum number of errors with the -maxErrors parameter, which is short for --log.maxErrors, and defaults to 10:
$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id --log.maxErrors 3
Example 16.2: Setting the logging directory
dsbulk logs to a directory named logs in the current directory where dsbulk is invoked. You can specify a different location with the -logDir parameter, which is short for --log.directory:
$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -logDir /tmp/logs
Example 16.3: Setting the logging verbosity
dsbulk has a way to dial up or down the verbosity of the logging, via --log.verbosity. Valid levels are 0, 1, and 2, from least to most verbose. For example, to get the minimum output:
$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id --log.verbosity 0
Example 17: Monitoring
dsbulk has a few ways to monitor the progress of dsbulk. The two main ways are reporting to the screen and via JMX.
Example 17.1: Setting the report rate
The progress is reported to the screen by default every 5 seconds. This rate can be set with the -reportRate parameter, which is a shortcut for --monitoring.reportRate:
$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -reportRate 2s
Or:
$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id -reportRate "2 seconds"
Example 17.2: Enabling/Disabling JMX
By default, dsbulk will provide metrics via JMX. You can disable this via the --monitoring.jmx parameter:
$ dsbulk load -url /tmp/dsbulkblog/iris.csv -k dsbulkblog -t iris_with_id --monitoring.jmx false
To download the DataStax Bulk Loader click here.
For an intro to unloading, read the next Bulk Loader blog here.
For basic loading examples, read the previous Bulk Loader blog here.
