Expanding a DataStax AMI Cluster¶

Recommendations¶

As a best practice when expanding a cluster, DataStax recommends doubling the size of the cluster size with each expansion. You should calculate the node number and token for each node based on the final ring size. For example, suppose that you ultimately want a 12 node cluster, starting with three node cluster. The initial three nodes are 0, 4, and 8. For the first expansion, you double the number of nodes to six by adding nodes, 2, 6, and 10. For the final expansion, you add six more nodes: 1, 3, 5, 7, 9, and 11. Using the tokengentool, calculate tokens for 12 nodes and enter the corresponding values in the initial_token property in the cassandra.yaml file. Adding Capacity to an Existing Cluster provides additional information about adding nodes and data centers.

Expanding a DataStax AMI Cluster Steps¶

1. In the AWS Management Console, create the number of nodes you need in another cluster with a temporary name. See Initializing a DataStax Enterprise Cluster on Amazon EC2.

   The temporary name prevents the new nodes from joining the cluster with the wrong tokens.

2. After the nodes have initialized, login to each node and stop the service:

   sudo service dse stop
   

3. Clear the data in each node:

   sudo rm -rf /var/lib/cassandra/*
   

   You must clear the data because new nodes have existing data from the initial start with the temporary cluster name and token.

4. For each node, change the /etc/dse/cassandra/cassandra.yaml settings to match the cluster_name and - seeds list of the other cluster. For example:

   cluster_name: 'NameOfExistingCluster'
   ...
   initial_token: 28356863910078205288614550619314017621
   ...
   seed_provider:
       - class_name: org.apache.cassandra.locator.SimpleSeedProvider
         parameters:
             - seeds: "110.82.155.0,110.82.155.3"
   

5. Assign the correct initital_token to each node; it determines the node's placement within the ring.

6. Set auto_bootstrap:

   • If adding nodes to an existing data center, set auto_bootstrap: true.
   • If adding nodes to a new data center, set auto_bootstrap: false, then run a rolling nodetool repair after all nodes have joined the ring.

7. Start each node in two minute intervals:

   sudo service dse start
   

8. Verify that each node has finished joining the ring:

   nodetool -h <hostname> ring
   

   
   
   

   Each node should show Up not Joining.