Apache Cassandra™ 2.0

Cassandra bulk loader (sstableloader)

The Cassandra bulk loader, also called the sstableloader tool, provides the ability to bulk load external data into a cluster, load existing SSTables into another cluster with a different number of nodes or replication strategy, and restore snapshots.

The sstableloader tool streams a set of SSTable data files to a live cluster. It does not simply copy the set of SSTables to every node, but transfers the relevant part of the data to each node, conforming to the replication strategy of the cluster. The table into which the data is loaded does not need to be empty.

If tables are repaired in a different cluster, after being loaded, the tables will be unrepaired.

Because sstableloader uses Cassandra gossip, make sure of the following:

  • The cassandra.yaml configuration file is in the classpath and properly configured.
  • At least one node in the cluster is configured as seed.
  • In the cassandra.yaml file, the following properties are properly configured for the cluster that you are importing into:

If you use sstableloader to load external data, you must first generate SSTables.

If you use DataStax Enterprise, you can use Sqoop to migrate external data to Cassandra.

Generating SSTables

SSTableWriter is the API to create raw Cassandra data files locally for bulk load into your cluster. The Cassandra source code includes the CQLSSTableWriter implementation for creating SSTable files from external data without needing to understand the details of how those map to the underlying storage engine. Import the org.apache.cassandra.io.sstable.CQLSSTableWriter class, and define the schema for the data you want to import, a writer for the schema, and a prepared insert statement, as shown in Cassandra 2.0.1, 2.0.2, and a quick peek at 2.0.3.

Using sstableloader

Before loading the data, you must define the schema of the tables with CQL or Thrift.

To get the best throughput from SSTable loading, you can use multiple instances of sstableloader to stream across multiple machines. No hard limit exists on the number of SSTables that sstablerloader can run at the same time, so you can add additional loaders until you see no further improvement.

If you use sstableloader on the same machine as the Cassandra node, you can't use the same network interface as the Cassandra node. However, you can use the JMX StorageService > bulkload() call from that node. This method takes the absolute path to the directory where the SSTables are located, and loads them just as sstableloader does. However, because the node is both source and destination for the streaming, it increases the load on that node. This means that you should load data from machines that are not Cassandra nodes when loading into a live cluster.

The sstableloader bulk loads the SSTables found in the directory dir_path to the configured cluster. The parent directory of dir_path is used as the keyspace name. For example to load an SSTable named Keyspace1-Standard1-he-1- Data.db into keyspace Keyspace1, the files Keyspace1-Standard1-he-1-Data.db and Keyspace1-Standard1-he-1-Index.db must be in a directory called Keyspace1/Standard1/.

Packaged installs:

$ sstableloader [options ] dir_path

Tarball installs:

$ cd install_location/bin
$ sstableloader [options ] dir_path

For example:

$ ls -1 Keyspace1/Standard1/
Keyspace1-Standard1-he-1-Data.db
Keyspace1-Standard1-he-1-Index
$ install_location/bin/sstableloader -d 110.82.155.1 Keyspace1/Standard1/

where dir_name is the directory containing the SSTables.

sstableloader
Short option Long option Description
-d <initial hosts> --nodes <initial hosts> Connect to comma separated list of hosts for initial ring information.
  -debug Display stack traces.
-h --help Display help.
-i <NODES> --ignore <NODES> Do not stream to this comma separated list of nodes.
  --no-progress Do not display progress.
-p <rpc port> --port <rpc port> RPC port (default 9160).
-t <throttle> --throttle <throttle> Throttle speed in Mbits (default unlimited).
-v --verbose Verbose output.
Show/hide