Apache Cassandra 1.0 Documentation

Initializing Multiple Data Center Clusters on Cassandra

This document corresponds to an earlier product version. Make sure you are using the version that corresponds to your version.

Latest Cassandra documentation | Earlier Cassandra documentation

In this scenario, data replication can be distributed across multiple, geographically dispersed data centers, between different physical racks in a data center, or between public cloud providers and on-premise managed data centers.

Data replicates across the data centers automatically and transparently; no ETL work is necessary to move data between different systems or servers. You can configure the number of copies of the data in each data center and Cassandra handles the rest, replicating the data for you. To configure a multiple data center cluster, see Initializing a Multiple Node Cluster in a Single Data Center.

Prerequisites

To correctly configure a multi-node cluster with multiple data centers, requires:

  • Cassandra is installed on each node.
  • The total number of nodes in the cluster.
  • A name for the cluster.
  • The IP addresses of each node in the cluster.
  • Which nodes will serve as the seed nodes. (Cassandra nodes use this host list to find each other and learn the topology of the ring.)
  • The snitch you plan to use.
  • If the nodes are behind a firewall, make sure you know what ports you need to open. See Configuring Firewall Port Access.
  • Other configuration settings you may need are described in Choosing Node Configuration Options and Node and Cluster Configuration.

This information is used to configure the following properties on each node in the cluster:

Configuration Example

This example describes installing a six node cluster spanning two data centers.

You set properties for each node in the cassandra.yaml file and the cassandra-topology.properties file. The location of these files depends on the type of installation; see Cassandra Configuration Files Locations or DataStax Enterprise Configuration Files Locations.

Note

After changing properties in these files, you must restart the node for the changes to take effect.

To configure a cluster with multiple data centers:

  1. Suppose you install Cassandra on these nodes:

    10.168.66.41 (seed1)
    10.176.43.66
    10.168.247.41
    10.176.170.59 (seed2)
    10.169.61.170
    10.169.30.138
  2. Assign tokens so that data is evenly distributed within each data center or replication group by calculating the token assignments with the Token Generating Tool and then offset the tokens for the second data center:

    Node IP Address Token Offset Data Center
    node0 10.168.66.41 0 NA DC1
    node1 10.176.43.66 56713727820156410577229101238628035242 NA DC1
    node2 10.168.247.41 113427455640312821154458202477256070485 NA DC1
    node3 10.176.170.59 10 10 DC2
    node4 10.169.61.170 56713727820156410577229101238628035252 10 DC2
    node5 10.169.30.138 113427455640312821154458202477256070495 10 DC2
  1. Stop the nodes and clear the data.

    • For packaged installs, run the following commands:

      $ sudo service cassandra stop (stops the service)

      $ sudo rm -rf /var/lib/cassandra/* (clears the data from the default directories)

    • For binary installs, run the following commands from the install directory:

      $ ps auwx | grep cassandra (finds the Cassandra Java process ID [PID])

      $ sudo kill <pid> (stops the process)

      $ sudo rm -rf /var/lib/cassandra/* (clears the data from the default directories)

  2. Modify the following property settings in the cassandra.yaml file for each node:

    • endpoint_snitch <name of snitch> - See endpoint_snitch.
    • initial_token: <token from previous step>
    • -seeds: <internal IP_address of each seed node>
    • listen_address: <localhost IP address>

    node0:

    end_point_snitch: org.apache.cassandra.locator.PropertyFileSnitch
    initial_token: 0
    seed_provider:
       - class_name: org.apache.cassandra.locator.SimpleSeedProvider
          parameters:
            - seeds: "10.168.66.41,10.176.170.59"
    listen_address: 10.176.43.66

    Note

    You must include at least one node from each data center. It is a best practice to have at more than one seed node per data center and the seed list should be the same for each node.

node1 to node5

The properties for the rest of the nodes are the same as Node0 except for the initial_token and listen_address:
  1. Determine a naming convention for each data center and rack, for example: DC1, DC2 or 100, 200 and RAC1, RAC2 or R101, R102.

  2. In the cassandra-topology.properties file, assign data center and rack names to the IP addresses of each node. For example:

    # Cassandra Node IP=Data Center:Rack
    10.168.66.41=DC1:RAC1
    10.176.43.66=DC2:RAC1
    10.168.247.41=DC1:RAC1
    10.176.170.59=DC2:RAC1
    10.169.61.170=DC1:RAC1
    10.169.30.138=DC2:RAC1
  3. Also, in the cassandra-topologies.properties file, assign a default data center name and rack name for unknown nodes.

    # default for unknown nodes
    default=DC1:RAC1
  4. After you have installed and configured Cassandra on all nodes, start the seed nodes one at a time, and then start the rest of the nodes.

    Note

    If the node has restarted because of automatic restart, you must stop the node and clear the data directories, as described above.

    • Packaged installs: sudo service cassandra start

    • Binary installs, run one of the following commands from the install directory:

      bin/cassandra (starts in the background)

      bin/cassandra -f (starts in the foreground)

  5. Check that your ring is up and running:

    • Packaged installs: nodetool ring -h localhost

    • Binary installs:

      cd /<install_directory>

      $ bin/nodetool ring -h localhost

    The ring status is displayed. This can give you an idea of the load balanced within the ring and if any nodes are down. If your cluster is not properly configured, different nodes may show a different ring; this is a good way to check that every node views the ring the same way.

Balancing the Data Center Nodes

When you deploy a Cassandra cluster, you need to use the partitioner to distribute roughly an equal amount of data to nodes. You also use identifiers for each data center (see step 5) in a formula to calculate tokens that balance nodes within a data center (DC). For example, assign each DC a numerical name that is a multiple of 100. Then for each DC, determine the tokens as follows: token = (2 127 / num_nodes_in_dc * n + DC_ID) where n is the node for which the token is being calculated and DC_ID is the numerical name.

You need to rebalance a data center after adding or removing a node. Nodes in other data centers do not have to be rebalanced. You need to stagger tokens between data centers for maximum data availability.

More Information About Configuring Data Centers

Links to more information about configuring a data center: