DataStax Enterprise 3.0 Documentation

Upgrading DSE Search/Solr nodes

This documentation corresponds to an earlier product version. Make sure this document corresponds to your version.

Latest DSE documentation | Earlier DSE documentation

To upgrade DataStax Enterprise 1.x.x - 2.2.x to 3.0.x, perform these upgrade steps on each node in the cluster. If the cluster is a mixed workload cluster, upgrade in the order described in Order of upgrading nodes. You need to restart the nodes as real-time Cassandra nodes before upgrading as described in this procedure. Restarting the nodes as real-time Cassandra nodes prevents unwanted schema changes from occurring when you start the upgraded node. Complete all steps on one node before starting to upgrade the next node.

Tarball release

  1. Stop the first node to be upgraded and restart it in real-time Cassandra mode:

    dse cassandra
    
  2. Create a directory for the new installation, download the tarball, and move it to that directory.

  3. Unpack the DataStax Enterprise 3.0.x tarball in the new install location.

    tar –xzvf <dse-3.0.x tarball name>
    
  4. If you customized the location of the data in the old installation, create a symbolic link to the old data directory:

    cd <new install location>
    ln -s <old data directory> <new install location>/<new data directory>
    

To configure the upgraded node

  1. In the new installation, open the cassandra.yaml for writing. The file is located in:

    <install location>/resources/cassandra/conf
    
  2. In the old installation of Cassandra, open the cassandra.yaml. The file is located in:

    <install location>/conf
    
  3. Diff the new and old cassandra.yaml files.

  4. Merge the diffs by hand from the old file into the new one, except do not merge snitch settings.

    If you are migrating data and set up the symbolic link described in the previous procedure, ensure that you merge the data_file_directories, commitlog_directory, and saved_caches_directory properties correctly.

  5. Configure the snitch setting.

  6. If you customized property files, other than the cassandra-topology.properties, update files by hand. Merge the settings of old property files, other than cassandra-topology.properties, into the new property files instead of overwriting the files. Users who overwrite property files, other than cassandra-topology.properties, have reported problems.

    It is ok to overwrite the old with the new cassandra-topology.properties file as instructed in Configuring the snitch setting.

  7. Start each node in real-time Cassandra mode during the rolling restart.

  8. Check for schema disagreements on each node.

  9. After all nodes are upgraded and the schemas agree, restart the nodes in Solr mode using rolling restart for Solr nodes that includes recovering the index.

  10. Update the old Solr configuration files and indexes.

  11. If you are upgrading from a version of DataStax Enterprise 3.0 or earlier, make the default legacy type mapping effective by commenting out the dseTypeMappingVersion element.

    <!-- <dseTypeMappingVersion>1</dseTypeMappingVersion> -->
    
  12. Repeat the previous steps for all nodes. Monitor the log files for any issues.

Packaged release

  1. Stop the dse service, and then disable Solr by setting the options in /etc/default/dse: SOLR_ENABLED=0

  2. Restart the dse service.

  3. Run the Yum (CentOS/RHEL/Oracle Linux) or Aptitude (Debian/Ubuntu) update commands.

  4. Run the install commands shown in Installing the DataStax Enterprise package on Debian and Ubuntu or Installing the DataStax Enterprise package on RHEL-based distributions.

  5. Start the first node.

  6. Configure the node: Open the old cassandra.yaml. Open the new cassandra.yaml:

    Debian/Ubuntu: /etc/dse/cassandra

    RHEL-based: /etc/dse/cassandra/cassandra.yaml

    Diff the new and old cassandra.yaml files. Merge the diffs by hand from the old file to the new one except do not merge the snitch setting.

  7. Configure the snitch setting.

  8. If you customized property files, other than the cassandra-topology.properties, update files by hand. Merge the settings of old property files, other than cassandra-topology.properties, into the new property files instead of overwriting the files. Users who overwrite property files, other than cassandra-topology.properties, have reported problems.

    It is ok to overwrite the old with the new cassandra-topology.properties file as instructed in Configuring the snitch setting.

  9. Start up each node in real-time Cassandra mode during the rolling restart (Solr mode disabled).

  10. Check for schema disagreements.

  11. After all nodes are upgraded and the schemas agree, reset the mode from real-time Cassandra to Solr.

To reset the mode to Solr

  1. Stop the dse service, and then enable Solr by setting this option in /etc/default/dse: SOLR_ENABLED=1

  2. Start the dse service using a rolling restart for Solr nodes that includes recovering the index.

  3. Update the old Solr configuration files and indexes.

  4. If you are upgrading from a version of DataStax Enterprise 3.0 or earlier, make the default legacy type mapping effective by commenting out the dseTypeMappingVersion element.

    <!-- <dseTypeMappingVersion>1</dseTypeMappingVersion> -->
    
  5. Repeat the previous steps for each node. Monitor the log files for any issues.

Performing a rolling restart of DSE Search/Solr nodes

To perform a rolling restart, restart each node in a rolling fashion, one node at a time, instead of bringing down the entire cluster and starting all nodes at once. If you are upgrading from a 2.x release, you likely will see error messages that resemble those shown in the list of Solr errors. If you get error messages, you typically recreate the core using the recovery option of the CREATE command. The recovery option is used when the core refuses to load due problems, such as incompatible changes during an upgrade or serious index errors. The default is false.

This diagram depicts the steps in the following procedure:


../../_images/rolling_restart.png

To restart the cluster using a rolling restart

  1. Restart the first node.

    If you do not see errors, proceed to step 2; otherwise, skip step 2 and do step 3 (run the CREATE OR RELOAD command).

  2. Check the Solr admin console to see if your cores are listed. If your cores are up, skip step 3 and do step 4 (check/resolve schema disagreements); otherwise, do step 3.

  3. Run the CREATE command to recover and not to distribute the core for each index.

    For example, to run the CREATE command for the demo, wikipedia, and log_search indexes, on the node you've just upgraded:

    curl -v "http://localhost:8983/solr/admin/cores?action=CREATE&name=demo.solr&recovery=true&distributed=false"
    
    curl -v "http://localhost:8983/solr/admin/cores?action=CREATE&name=wiki.solr&recovery=true&distributed=false"
    
    curl -v "http://localhost:8983/solr/admin/cores?action=CREATE&name=Logging.log_entries&recovery=true&distributed=false"
    

    If you get an error message telling you that the core already exists, run the RELOAD command instead of the CREATE command.

  4. Check for and resolve schema disagreements for the node. Monitor the log files for any issues.

  5. If you use counters or Cassandra 1.1 or earlier, upgrade SSTables.

  6. Repeat the procedure for the next node.

Expected Solr error messages during the upgrade

When installing DataStax Enterprise 3.0.x, you may see errors, such as:

ERROR 00:57:17,785 Cannot activate core: ks.cf_10000_keys_50_cols
ERROR 00:57:17,786 <indexDefaults> and <mainIndex> configuration sections are discontinued.
    Use <indexConfig> instead.

ERROR 01:29:55,145 checksum mismatch in segments file (resource:
   ChecksumIndexInput(MMapIndexInput(path="/var/lib/cassandra/data/solr.data/ks.
   cf_10000_keys_50_cols/index/segments_6")))
ERROR 01:29:55,145 Solr index ks.cf_10000_keys_50_cols seems to be corrupted:
   please CREATE the core again with recovery=true to start reindexing data.
ERROR 01:29:55,145 Cannot activate core: ks.cf_10000_keys_50_cols
ERROR 01:29:55,146 checksum mismatch in segments file (resource: ChecksumIndexInput
   (MMapIndexInput(path="/var/lib/cassandra/data/solr.data/ks.
   cf_10000_keys_50_cols/index/segments_6")))
org.apache.lucene.index.CorruptIndexException: checksum mismatch in segments file
   (resource: ChecksumIndexInput(MMapIndexInput
   (  path="/var/lib/cassandra/data/solr.data/ks.cf_10000_keys_50_cols/index/segments_6")))