Alex Ott

Sebastian Estevez

<p>When we launched Astra, the multi-cloud Database-as-a-Service (DBaaS) for Apache Cassandra® applications, we also <a href="https://www.datastax.com/blog/2020/05/why-astra-good-cassandra">open-sourced portions of the architecture</a> and tooling backing it. Today, we continue our promise to lead with code. Your first <a href="https://www.datastax.com/blog/2020/07/take-flight-live-and-free-migration-your-apache-cassandra-apps">Cassandra application migration</a> to Astra is free, and knowing how to migrate additional Cassandra applications is certainly no secret.&nbsp;</p>

<h2>One-time Cassandra cluster migration</h2>

<p>One-time migration of data from your own Cassandra cluster to <a href="https://astra.datastax.com/">Astra</a> is the simplest way to migrate an application. (A <a href="https://www.datastax.com/migration/astra-sizing">Cassandra migration tool</a> is currently available for early access. Sign up and try it out!)</p>

<p>In the meantime, we’ll guide you through the migration in four steps:</p>

<ol>
	<li>Create your Keyspace:Table in the destination cluster and download the connection bundle.</li>
	<li>Locally, install <a href="https://spark.apache.org/">Apache Spark</a> and the new version of <a href="https://github.com/datastax/spark-cassandra-connector/tree/b2.5">DataStax's Spark Cassandra Connector</a> (open-source).</li>
	<li>Use the DataFrames API to push your data to the remote Keyspace: Table.</li>
	<li>Verify your data on the destination cluster.</li>
</ol>

<h3>Step 1: Create your Keyspace:Table</h3>

<p>In this example, we’ll use a table from a popular workload in <a href="http://docs.nosqlbench.io/">NoSQLBench</a> in the source cluster. The table has the following structure:</p>

<pre>
<code>
CREATE TABLE baselines.iot (
    machine_id uuid,
    sensor_name text,
    time timestamp,
    data text,
    sensor_value double,
    station_id uuid,
    PRIMARY KEY ((machine_id, sensor_name), time)
) WITH CLUSTERING ORDER BY (time DESC);
</code>
</pre>

<p>To migrate data, create a corresponding table in the destination cluster in Astra, which could be done directly in Astra's UI, via the "CQL Console" tab. (I've created it with the name test.iot.)</p>

<p>To connect to a destination database, download your access credentials as a secure connect bundle for your Cassandra cluster created on DataStax Astra. (For me, it was stored as ~/secure-connect-test.zip.)</p>

<h3>Step 2: Install Spark and Cassandra Connector</h3>

<p>After the secure bundle is downloaded, start Spark with the Spark Cassandra Connector. Your Spark configuration could be as simple as a single Spark node in local master mode. I'm using Spark 2.4.5, compiled with Scala 2.11, so my command line looks like the following:</p>

<pre>
<code>
bin/spark-shell --packages com.datastax.spark:spark-cassandra-connector_2.11:2.5.1 \
  --files ~/secure-connect-test.zip
</code>
</pre>

<h3>Step 3: Push your data to the remote Keyspace:Table</h3>

<p>After the Spark Shell has started, a standard DataFrame API can be used to read the data from the source cluster. In my case, the contact point is 10.101.34.176. To write data to the destination cluster in Astra, provide a filename with the secure connect bundle, as well as the username and password:</p>

<pre>
<code>
val df = spark
  .read
  .format("org.apache.spark.sql.cassandra")
  .options(Map(
    "keyspace" -&gt; "baselines",
    "table" -&gt; "iot",
    "spark.cassandra.connection.host" -&gt; "10.101.34.176"
  ))
  .load

df.write
  .format("org.apache.spark.sql.cassandra")
  .options(Map(
    "keyspace" -&gt; "test",
    "table" -&gt; "iot",
    "spark.cassandra.connection.config.cloud.path" -&gt; "secure-connect-test.zip",
    "spark.cassandra.auth.password" -&gt; "123456", 
    "spark.cassandra.auth.username" -&gt; "test"
  ))
  .save
</code>
</pre>

<h3>Step 4: Verify data</h3>

<p>That's all! We can check that the data successfully copied by comparing the number of rows in both tables:</p>

<pre>
<code>
val srcCount = df.count
val destCount = spark.read.format("org.apache.spark.sql.cassandra")
  .options(Map(
    "keyspace" -&gt; "test",
    "table" -&gt; "iot",
    "spark.cassandra.connection.config.cloud.path" -&gt; "secure-connect-test.zip",
    "spark.cassandra.auth.password" -&gt; "123456", 
    "spark.cassandra.auth.username" -&gt; "test"
  )).count 
println("source count" + srcCount)
println("destination count" + destCount)
</code>
</pre>

<p>Try this yourself by using <a href="https://github.com/datastax/spark-cassandra-connector/tree/b2.5">Spark Cassandra Connector 2.5.</a>1 with <a href="https://spark.apache.org/">Apache Spark</a> and easily migrate your existing database to <a href="https://astra.datastax.com/">DataStax Astra</a>!</p>

<h2>Alternative Option: Using the RDD API</h2>

<p>Of course, we can also use the RDD API to perform a migration, but, as you can see, it's more verbose:</p>

<pre>
<code>
import com.datastax.spark.connector._
import com.datastax.spark.connector.cql._
val sourceConnection = CassandraConnector(sc.getConf.set("spark.cassandra.connection.host", "10.101.34.176"))
val destConnection = CassandraConnector(sc.getConf
  .set("spark.cassandra.connection.config.cloud.path", "secure-connect-test.zip")
  .set("spark.cassandra.auth.username", "test")
  .set("spark.cassandra.auth.password", "123456"))

val rdd = {
  implicit val c = sourceConnection
  sc.cassandraTable("baselines","iot")
}

{
  implicit val c = destConnection
  rdd.saveToCassandra("test","iot")
}
</code>
</pre>

<h2>Migrating data from Cassandra to Astra DBaaS is no mystery</h2>

<p>We hope this simple, four-step tutorial provides solid ground for migrating your Cassandra data to Astra DBaaS. Things might get a little messier, if you decide to use the RDD API, but at least you know you have options.&nbsp;</p>

<p>Of course, not all application migrations are easy. But, don’t worry. Our DataStax team is here to help. In fact, <a href="https://www.datastax.com/blog/take-flight-live-and-free-migration-your-apache-cassandra-apps">migration of your first Cassandra application is completely free</a>. It’s on us with no catches.</p>

<p>Or, <a href="https://www.datastax.com/products/datastax-astra">learn more about Astra DB</a>.</p>


Tutorial: Cassandra Migration to Astra DBaaS

Alex Ott

Sebastian Estevez

Share

More Technology

Getting Started with DataStax Astra DB and Amazon Bedrock

Why You Can't Miss Cassandra Summit

Mitigate the OpenAI Debacle with Open Source and Cloud

Supercharge AI Assistants with Superagent and DataStax

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale

Alex Ott

Sebastian Estevez

Share

More Technology

Getting Started with DataStax Astra DB and Amazon Bedrock

Why You Can't Miss Cassandra Summit

Mitigate the OpenAI Debacle with Open Source and Cloud

Supercharge AI Assistants with Superagent and DataStax

NoSQL and Vector DBfor Generative AI,Instantly, At Scale

NoSQL and Vector DB
for Generative AI,
Instantly, At Scale