Brian Hess

Earlier this week, we made both the <a href="https://www.datastax.com/dev/datastax-bulk-loader">DataStax Bulk Loader</a> and <a href="https://www.datastax.com/dev/datastax-apache-kafka-connector">DataStax Apache Kafka®</a> Connector tools freely available to any developer working with open source Cassandra. What better time to conclude our DataStax Bulk Loader blog series, with a look at examples for loading from other locations.&nbsp;

There are a number of cases where we want to load data from other locations into DataStax Enterprise (DSE), including:

<h2>Example 30: Load from another DSE cluster</h2>

One common use case is moving data from one DSE cluster to another. dsbulk does not have a “migrate” method, but we can accomplish this with native load and unload support. One way to do this is to unload data from one cluster to the local file system and then load that data into the other cluster: 
<code>$ dsbulk unload -h localhost -k dsbulkblog -t iris_with_id -url /tmp/dsbulkblog/migrate 
$ dsbulk load -h localhost -k dsbulkblog -t iris_with_id -url /tmp/dsbulkblog/migrate</code>

One nifty way to do this is in one go, piping the output of<code> dsbulk unload to<code> dsbulk load: 
$ dsbulk unload -h localhost -k dsbulkblog -t iris_with_id | dsbulk load -h localhost -k dsbulkblog -t iris_with_id</code></code>

This gives the following (interleaved) output: 
<code><code>Operation directory: /tmp/logs/LOAD_20190314-172058-173447. 
Operation directory: /tmp/logs/UNLOAD_20190314-172058-192179. 
total | failed | rows/s | mb/s | kb/row | p50 ms | p99ms | p999ms 
&nbsp; 150 | &nbsp; &nbsp; &nbsp;0 | &nbsp; &nbsp;302 | 0.01 | &nbsp; 0.04 | &nbsp;20.12 | 20.19 | &nbsp;20.19 
Operation UNLOAD_20190314-172058-192179 completed successfully in 0 seconds. 
total | failed | rows/s | mb/s | kb/row | p50 ms | p99ms | p999ms | batches 
&nbsp; 150 | &nbsp; &nbsp; &nbsp;0 | &nbsp; &nbsp; 52 | 0.00 | &nbsp; 0.04 | &nbsp;20.94 | 60.29 | &nbsp;61.34 | &nbsp; &nbsp;1.00 
Operation LOAD_20190314-172058-173447 completed successfully in 2 seconds. 
Last processed positions can be found in positions.txt</code></code>

While this second way is nifty, it does have one downside that we should call out. The unload and load will both be single-threaded since they are going via stdout/stdin. Doing things in two steps is safer and will be done in parallel, but you will do all of your unloading before loading.

<h2>Example 30.1: Migrating while preserving TTLs and Timestamps</h2>

Sometimes we want to migrate the data while preserving the writetime and the TTL. We can do that via custom CQL queries with dsbulk. First, we unload the data with the writetime and TTL information:

<code><code>$ dsbulk unload -h localhost -query "SELECT id, petal_length, WRITETIME(petal_length) AS w_petal_length, TTL(petal_length) AS l_petal_length, petal_width, WRITETIME(petal_width) AS w_petal_width, TTL(petal_width) AS l_petal_width, sepal_length, WRITETIME(sepal_length) AS w_sepal_length, TTL(sepal_length) AS l_sepal_length, sepal_width, WRITETIME(sepal_width) AS w_sepal_width, TTL(sepal_width) AS l_sepal_width, species, WRITETIME(species) AS w_species, TTL(species) AS l_species FROM dsbulkblog.iris_with_id" -url /tmp/dsbulkblog/migrate</code></code>

The next thing to do is to load. We’re going to accomplish this by using batch statements. The reason is that we can only set one writetime and one TTL per INSERT statement. So, we will do one INSERT statement per regular column (petal_length, petal_width, sepal_length, sepal_width, and species). We will put all of these INSERTs into the same batch all keyed on the same partition key, namely id, so these will be collapsed in Cassandra into a single mutation and applied atomically:

<code>$ dsbulk load -h localhost -query "BEGIN BATCH INSERT INTO dsbulkblog.iris_with_id(id, petal_length) VALUES (:id, :petal_length) 
USING TIMESTAMP :w_petal_length AND TTL :l_petal_length; INSERT INTO dsbulkblog.iris_with_id(id, petal_width) VALUES (:id, :petal_width) 
USING TIMESTAMP :w_petal_width AND TTL :l_petal_width; INSERT INTO dsbulkblog.iris_with_id(id, sepal_length) VALUES (:id, :sepal_length) 
USING TIMESTAMP :w_sepal_length AND TTL :l_sepal_length; INSERT INTO dsbulkblog.iris_with_id(id, sepal_width) VALUES (:id, :sepal_width) 
USING TIMESTAMP :w_sepal_width AND TTL :l_sepal_width; INSERT INTO dsbulkblog.iris_with_id(id, species) VALUES (:id, :species) 
USING TIMESTAMP :w_species AND TTL :l_species; APPLY BATCH;" -url /tmp/dsbulkblog/migrate --batch.mode DISABLED</code>

<h2>Example 30.2: Migrating within the same cluster</h2>

The above examples can be used when the source and destination clusters are the same. One good example is if you need to change the schema of the table and it already has data in it. While adding (or removing) a regular column is simple and supported in CQL, changing the primary keys (partition keys or clustering columns) is not supported. To accomplish that, you need to copy the data, something that is also not supported in CQL.

For example, let’s take our iris data and change the schema to make the species the partition key and the ID column the clustering column. We can do that via CQL: 
<code>$ cqlsh -e "CREATE TABLE dsbulkblog.iris_by_species(id INT, petal_length DOUBLE, petal_width DOUBLE, sepal_length DOUBLE, sepal_width DOUBLE, species TEXT, PRIMARY KEY ((species), id));"</code>

Now, we can unload the data from the iris_with_id table and load that data into iris_by_species table: 
<code>$ dsbulk unload -h localhost -k dsbulkblog -t iris_with_id | dsbulk load -h localhost -k dsbulkblog -t iris_by_species</code>

We can spot-check that the data has migrated by comparing the results of a dsbulk count: 
<code>$ dsbulk count -k dsbulkblog -t iris_with_id -stats global --log.verbosity 0 
150 
$ dsbulk count -k dsbulkblog -t iris_by_species -stats global --log.verbosity 0 
150</code>

<h2>Example 31: Loading from DSEFS</h2>

One common use case to consider is how to load data from DSEFS. dsbulk does not have native support for loading from DSEFS URLs. &nbsp;That said, there are a couple of ways we can address this. &nbsp;To demonstrate this, on a cluster with Analtyics enabled (and thus DSEFS enabled) let’s first put some data in DSEFS: 
<code>$ dse fs "mkdir dsefs:///dsbulkblog" 
$ dse fs "cp file:///tmp/dsbulkblog/iris.csv dsefs:///dsbulkblog/"</code>

We can see that the data is now in there with: 
<code>$ dse fs "cat dsefs:///dsbulkblog/iris.csv"</code>

Now, we can use that same command and pipe it into dsbulk: 
<code>$ dse fs "cat dsefs:///dsbulkblog/iris.csv" | dsbulk load -k dsbulkblog -t iris_with_id</code>

We could also do this with the WebHDFS interface to DSEFS: 
<code>$ dsbulk load -k dsbulkblog -t iris_with_id -url &nbsp;http://localhost:5598/webhdfs/v1/dsbulkblog/iris.csv?op=OPEN</code>

<h2>Example 32: Loading from MySQL</h2>

In a similar fashion, we can migrate data from an RDBMS to DSE by combining the command-line utility of the RDBMS to write data to its stdout and then pipe that into dsbulk. &nbsp;For MySQL, for example, it would look something like this (Note: MySQL will export in tab-delimited form): 
<code>$ mysql --user=mysqluser --password=mypassword -B --execute="SELECT id, petal_length, petal_width, sepal_length, sepal_width, species FROM mydb.iris_with_id" | dsbulk load -k dsbulkblog -t iris_with_id -delim "\t" -header false -m "id,petal_length,petal_width,sepal_length,sepal_width,species"</code>

<h2>Wrap-up</h2>

As part of our ongoing support of the Cassandra community, we were very excited to make our <a href="https://www.datastax.com/dev/datastax-apache-kafka-connector">DataStax Apache Kafka™ Connector</a> and <a href="https://www.datastax.com/dev/datastax-bulk-loader">DataStax Bulk Loader</a> for Cassandra announcement. Read about the free tools in this <a href="https://www.datastax.com/blog/2019/12/tools-apache-cassandratm">blog post</a>.

If you want to learn more about DataStax Bulk Loader, here are the previous blog posts in this series:&nbsp;

<ul>
	<li><a href="https://www.datastax.com/blog/2019/03/datastax-bulk-loader-introduction-and-loading">DataStax Bulk Loader: Introduction and Loading</a></li>
	<li><a href="https://www.datastax.com/blog/2019/04/datastax-bulk-loader-more-loading">DataStax Bulk Loader: More Loading</a></li>
	<li><a href="https://www.datastax.com/blog/2019/04/datastax-bulk-loader-common-settings">DataStax Bulk Loader: Common Settings</a></li>
	<li><a href="https://www.datastax.com/blog/2019/06/datastax-bulk-loader-unloading">DataStax Bulk Loader: Unloading</a></li>
	<li><a href="https://www.datastax.com/blog/2019/07/datastax-bulk-loader-counting">DataStax Bulk Loader: Counting</a> 
	&nbsp;</li>
</ul>

DataStax Bulk Loader Part 6 - Examples for Loading From Other Locations

Brian Hess

Discover more

Share

Share

Example 30: Load from another DSE cluster

Example 30.1: Migrating while preserving TTLs and Timestamps

Example 30.2: Migrating within the same cluster

Example 31: Loading from DSEFS

Example 32: Loading from MySQL

Wrap-up

More Technology

Knowledge Graphs for RAG without a GraphDB

How Winweb Built its AI Assistant with DataStax Astra DB and LangChain

Vercel + Astra DB: Get Data into Your GenAI Apps Fast

Simplifying Agent Development with Astra DB Connector for Vertex AI Search

One-stop Data API for Production GenAI