This section contains instructions for expanding a cluster that uses the DataStax Community Edition AMI (Amazon Machine Image) prior to Cassandra 1.2. If your AMI uses Cassandra 1.2, see the 1.2 instructions.
As a best practice when expanding a cluster, DataStax recommends doubling the size of the cluster size with each expansion. You should calculate the node number and token for each node based on the final ring size. For example, suppose that you ultimately want a 12 node cluster, starting with three node cluster. The initial three nodes are 0, 4, and 8. For the first expansion, you double the number of nodes to six by adding nodes, 2, 6, and 10. For the final expansion, you add six more nodes: 1, 3, 5, 7, 9, and 11. Using the tokengentool, calculate tokens for 12 nodes and enter the corresponding values in the initial_token property in the cassandra.yaml file. For more information, see Adding Capacity to an Existing Cluster.
In the AWS Management Console, create the number of nodes you need in another cluster with a temporary name. See Installing a Cassandra Cluster on Amazon EC2.
The temporary name prevents the new nodes from joining the cluster with the wrong tokens.
After the nodes are initialized, login to each node and stop the service:
sudo service cassandra stop
Clear the data in each node:
Check the cassandra.yaml for the location of the data directories:
data_file_directories: - /raid0/cassandra/data
Remove the data directories:
sudo rm -rf /raid0/cassandra/*
You must clear the data because new nodes have existing data from the initial start with the temporary cluster name and token.
For each node, change the /etc/dse/cassandra/cassandra.yaml settings to match the cluster_name and - seeds list of the other cluster. For example:
cluster_name: 'NameOfExistingCluster' ... initial_token: 28356863910078205288614550619314017621 ... seed_provider: - class_name: org.apache.cassandra.locator.SimpleSeedProvider parameters: - seeds: "18.104.22.168,22.214.171.124"
Assign the correct initital_token to each node; it determines the node's placement within the ring.
If adding nodes to an existing data center:
If adding nodes to a new data center:
Set auto_bootstrap: false.
After all new nodes have joined the ring, do either of the following:
$ nodetool repair -h 10.46.123.12 keyspace_name -pr
Start each node in two minute intervals:
$ sudo service cassandra start
Verify that each node has finished joining the ring:
nodetool -h <hostname> ring
Each node should show Up not Joining.