Between the time the first node in a cluster begins the upgrade process until the last node completes the process, a schema disagreement condition exists. Cassandra throws a SchemaDisagreementException when a schema disagreement occurs. This is expected behavior.
When the schema disagreement exists, client interfaces block the following operations:
DDL, TRUNCATE, and Solr queries are not supported during a rolling restart. For example, during a rolling restart, these are the CQL commands that are and are not supported:
|OK to Run||Do Not Run|
|DELETE||ALTER TABLE||DROP TABLE|
|INSERT||CREATE TABLE||DROP INDEX|
|SELECT||CREATE INDEX||DROP KEYSPACE|
Perform these steps to resolve the schema disagreement:
Using the Command Line Interface (CLI), run the DESCRIBE CLUSTER command. For example:
$ cassandra-cli -host localhost -port 9160 [default@unknown] DESCRIBE cluster;
If any node is UNREACHABLE, you see output something like this:
[default@unknown] describe cluster; Cluster Information: Snitch: com.datastax.bdp.snitch.DseDelegateSnitch Partitioner: org.apache.cassandra.dht.Murmur3Partitioner Schema versions: UNREACHABLE: [10.202.205.203, 10.80.207.102, 10.116.138.23]
Restart the unreachable nodes.
Repeat the previous steps until the DESCRIBE cluster command shows that all nodes have the same schema version number: only one schema version appears in the output of DESCRIBE cluster.
If you created tables having the CompactionStrategy configured to be LeveledCompactionStrategy, you need to scrub the SSTables that store those tables.
Before scrubbing the SSTables, upgrade all the nodes.
To scrub SSTables:
Shut down the nodes, one-at-a-time.
On each offline node, run the sstablescrub utility.
For example, on a tarball installation:
cd <install directory>/bin ./sstablescrub mykeyspace mytable
To get help about sstablescrub:
usage: sstablescrub -h
If you do not scrub the affected SSTables, you might encounter the following error during compactions on tables using LeveledCompactionStrategy:
ERROR [CompactionExecutor:150] 2012-07-05 04:26:15,570 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[CompactionExecutor:150,1,main] java.lang.AssertionError at org.apache.cassandra.db.compaction.LeveledManifest.promote (LeveledManifest.java:214)