DataStax Enterprise 3.1 Documentation

Upgrading to version 3.1.x

This documentation corresponds to an earlier product version. Make sure this document corresponds to your version.

Latest DSE documentation | Earlier DSE documentation

Version restrictions

Upgrading directly from versions other than these does not work:

  • DataStax Enterprise 2.2 or later
  • Cassandra 1.1.5 - 1.2.10
  • DataStax Community 1.1 (Cassandra 1.1.9)

To upgrade from earlier versions, first upgrade to DataStax Enterprise 2.2, and then upgrade the SSTables.

To upgrade from a Brisk release, contact Support.

Partitioner

Merge your partitioner setting from the old to the new file. Do not attempt to use the Cassandra 1.2 default partitioner option, Murmur3Partitioner, in the new file unless you were already using it.

CQL 3

Do not issue any CQL 3 queries until all nodes are upgraded and schema disagreements are resolved.

Security

The client_encryption_options for enabling client-to-node SSL have been removed from dse.yaml in 3.1.2 and later. To enable client-to-node SSL, set the option in the cassandra.yaml file.

Before upgrading, if you use these DataStax Enterprise security features, adjust the replication strategy and options in the cassandra.yaml file to configure a replication factor for the dse_auth keyspace greater than 1:

  • Kerberos
  • Object permission management (internal authorization)
  • Internal authentication

Adjust the replication factor for dse_auth on each node in the cluster. After updating the cassandra.yaml file and restarting the node, run nodetool repair to repair the first range returned by the partitioner for the keyspace:

nodetool repair dse_auth -pr

This should only take a few seconds to complete.

The new version of Cassandra updates the security options. First simply merge the following settings into the new configuration files:

  • authenticator
  • authorizer
  • auth_replication_strategy
  • auth_replication_options
  • any other diffs

Use the old settings while you are upgrading the cluster so that backward compatibility is maintained. For example, the new file contains the old, Cassandra 1.1 authenticator and authorizer options at this point:

  • authenticator: com.datastax.bdp.cassandra.auth.PasswordAuthenticator
  • authorizer: org.apache.cassandra.auth.CassandraAuthorizer

If you are upgrading a secure cluster, there may be a significant delay to each node's first startup as the security migration takes place (up to 1 minute). The delay is due to ensuring that the ring is fully connected before the migration starts. During the upgrade of a secure cluster, you may see a security related error message (documented below). You will see the following message in the log when the node has completed the migration:

INFO [NonPeriodicTasks:1] 2013-06-22 15:01:08,173
Auth.java (line 208) Migration of legacy auth data is complete.
You should now switch to org.apache.cassandra.auth implementations in cassandra.yaml.

After all nodes have been upgraded, change these options to the new Cassandra 1.2 values and perform a rolling restart as explained below.

Note

If using Kerberos authentication, there are no credentials data to migrate, but user records must still be updated. Merge the related diffs from the old to the new file.

  1. Edit the cassandra.yaml to switch to the official Apache versions of PasswordAuthenticator and CassandraAuthorizer:

    authenticator: org.apache.cassandra.auth.PasswordAuthenticator
    authorizer: org.apache.cassandra.auth.CassandraAuthorizer
    
  2. Remove these options from the cassandra.yaml file:

    • auth_replication_strategy
    • auth_replication_options
  3. Optionally, adjust the replication factor of the system_auth keyspace. The amount of data in this keyspace is typically very small, so leaving it replicated across the cluster is relatively cheap.

SSTable upgrades

After restarting each node, consider upgrading sstables. Upgrading SSTables is highly recommended under these conditions:

  • If you use counter columns
  • If you are upgrading from Cassandra 1.1.x or earlier
  • If you are upgrading from a DataStax Enterprise version having Cassandra 1.1.x or earlier

Upgrade SSTables before doing these operations:

  • move
  • repair
  • bootstrap

Because these operations copy SSTables within the cluster and the on-disk format sometimes changes between major versions, DataStax recommends upgrading SSTables now to prevent possible future SSTable incompatibilities:

  • Tarball: <install_location>/bin/nodetool -h upgradesstables
  • Package or AMI: nodetool -h upgradesstables

Virtual nodes

DataStax recommends using virtual nodes only on data centers running purely Cassandra workloads. You should disable virtual nodes on data centers running either Hadoop or Solr workloads by setting num_tokens to 1 in the cassandra.yaml.

Solr

If you are upgrading a Solr node, make the default legacy type mapping effective by commenting out the dseTypeMappingVersion element:

<!-- <dseTypeMappingVersion>1</dseTypeMappingVersion> -->

Expected error messages

If you are upgrading from DataStax Enterprise 3.0.x, an exception that looks something like this might appear in logs during a rolling upgrade. Ignore these error messages:

ERROR 15:36:54,908 Exception in thread Thread[GossipStage:1,5,main]
 java.lang.NumberFormatException: For input string: "127605887595351923798765477786913079296"
. . .

When upgrading Cassandra 1.2 nodes, you can ignore the following error messages related to when a node is attempting to push mutations to the new system_auth keyspace:

ERROR [WRITE-/192.168.123.11] 2013-06-22 14:13:42,336 OutboundTcpConnection.java (line 222)
 error writing to /192.168.123.11
java.lang.RuntimeException: Can't serialize ColumnFamily ID 2d324e48-3275-3517-8dd5-9a2c5b0856c5
to be used by version 5, because int <-> uuid mapping could not be established
(CF was created in mixed version cluster).
at org.apache.cassandra.db.ColumnFamilySerializer.cfIdSerializedSize(ColumnFamilySerializer.java:196)

When upgrading a Solr node, you can ignore the following error:

ERROR 00:57:17,785 Cannot activate core: ks.cf_10000_keys_50_cols
ERROR 00:57:17,786 <indexDefaults> and <mainIndex> configuration sections are discontinued.
 Use <indexConfig> instead.

ERROR 01:29:55,145 checksum mismatch in segments file (resource:
  ChecksumIndexInput(MMapIndexInput(path="/var/lib/cassandra/data/solr.data/ks.
  cf_10000_keys_50_cols/index/segments_6")))
ERROR 01:29:55,145 Solr index ks.cf_10000_keys_50_cols seems to be corrupted:
  please CREATE the core again with recovery=true to start reindexing data.
ERROR 01:29:55,145 Cannot activate core: ks.cf_10000_keys_50_cols
ERROR 01:29:55,146 checksum mismatch in segments file (resource: ChecksumIndexInput
  (MMapIndexInput(path="/var/lib/cassandra/data/solr.data/ks.
  cf_10000_keys_50_cols/index/segments_6")))
org.apache.lucene.index.CorruptIndexException: checksum mismatch in segments file
  (resource: ChecksumIndexInput(MMapIndexInput
  (  path="/var/lib/cassandra/data/solr.data/ks.cf_10000_keys_50_cols/index/segments_6")))

Recommissioning a node

If you decommissioned a node in the last 72 hours:

  • Do not recommission the node until 72 hours has passed.
  • If you wish to recommission the node after 72 hours, run "nodetool gossipinfo". Check the STATUS line for the token of the decommissioned node and verify that it does not exist. If it does not exist, then the node has been deleted and it is safe to recommission the node.
  • If you need to bring the node into the cluster, contact Support for detailed information on how to assassinate the node.