Apache Cassandra 1.2 Documentation

Replacing or adding a node or data center

This section discusses the following tasks:

Adding nodes an existing cluster

Virtual nodes greatly simplify adding nodes to an existing cluster:

  • Calculating tokens and assigning them to each node is no longer required.
  • Rebalancing a cluster is no longer necessary because a node joining the cluster assumes responsibility for an even portion of the data.

For a detailed explanation about how this works, see Virtual nodes in Cassandra 1.2.

Note

If you do not use virtual nodes, follow the instructions in the 1.1 topic Adding Capacity to an Existing Cluster.

To add nodes to a cluster

  1. Install Cassandra on the new nodes, but do not start Cassandra. (If you used a packaged install, Cassandra starts automatically and you must stop the node and clear the data.)
  1. Set the following properties in the cassandra.yaml and cassandra-topology.properties configurtion files:
    • cluster_name: The name of the cluster the new node is joining.
    • listen_address/broadcast_address: The IP address or host name that other Cassandra nodes use to connect to the new node.
    • endpoint_snitch: The snitch Cassandra uses for locating nodes and routing requests.
    • num_tokens: The number of virtual nodes to assign to the node. If the hardware capabilities vary among the nodes in your cluster, you can assign a proportional number of virtual nodes to the larger machines.
    • seed_provider: The - seeds list in this setting determines which nodes the new node should contact to learn about the cluster and establish the gossip process.
    • Change other non-default settings you have made to your existing cluster in the cassandra.yaml file and cassandra-topology.properties files. Use the diff command to find and merge (by head) any differences between existing and new nodes.
  2. Start Cassandra on each new node. Allow two minutes between node initializations. You can monitor the startup and data streaming process using nodetool netstats.
  3. After all new nodes are running, run nodetool cleanup on each of the previously existing nodes to remove the keys no longer belonging to those nodes. Wait for cleanup to complete on one node before doing the next.
Cleanup may be safely postponed for low-usage hours.

Adding a data center to a cluster

The following steps describe adding a data center to an existing cluster.

  1. Ensure that you are using NetworkTopologyStrategy for all of your keyspaces.

  2. For each new node, edit the configuration properties in the cassandra.yaml file:

    • Add (or edit) auto_bootstrap: false. By default, this setting is true and not listed in the cassandra.yaml file. Setting this parameter to false prevent the new nodes from attempting to get all the data from the other nodes in the data center. When you run nodetool rebuild in the last step, each node is properly mapped.
    • Set the necessary properties as described in step 2 above.

  3. If using the PropertyFileSnitch, update the cassandra-topology.properties file on all servers to include the new nodes. You do not need to restart.

    The location of this file depends on the type of installation; see Cassandra Configuration Files Locations or DataStax Enterprise Configuration Files Locations.

  4. Ensure that your client does not auto-detect the new nodes so that they aren't contacted by the client until explicitly directed. For example in Hector, set hostConfig.setAutoDiscoverHosts(false);

  5. If using a QUORUM consistency level for reads or writes, check the LOCAL_QUORUM or EACH_QUORUM consistency level to see if the level meets your requirements for multiple data centers.

  6. Start the new nodes.

  7. After all nodes are running in the cluster:

    1. Change the strategy_options for your keyspace to the desired replication factor for the new data center. For example: set strategy options to DC1:2, DC2:2. For more information, see ALTER KEYSPACE.
    2. Run nodetool rebuild on all nodes in the new data center.

Replacing a dead node

To replace a node that has died for some reason, such as hardware failure, prepare and start the replacement node, integrate it into the cluster, and then remove the dead node.

To prepare the replacement node:

  1. Confirm that the node is dead by running nodetool status (if using virtual nodes) or nodetool ring.

    The nodetool command shows a down status for the dead node:


    ../../_images/nodetool_status_arrow.png

  2. Add and start the replacement node as described in the steps above.

  3. Remove the dead node from the cluster using the removenode command. Use the force option of this command if the node is not removed.