DataStax Support Forums » Topic: Schema versions disagree

DataStax Support Forums » Topic: Schema versions disagree http://www.datastax.com/support-forums/topic/schema-versions-disagree Software, Support, and Training for Apache Cassandra en-US Wed, 19 Jun 2013 20:10:08 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php schalla on "Schema versions disagree" http://www.datastax.com/support-forums/topic/schema-versions-disagree#post-7065 Mon, 22 Oct 2012 22:07:24 +0000 schalla 7065@http://www.datastax.com/support-forums/ Apparently the servers were drifted from time sync. After the time sync, I no longer see "Schema disagreement". schalla on "Schema versions disagree" http://www.datastax.com/support-forums/topic/schema-versions-disagree#post-7059 Mon, 22 Oct 2012 17:51:09 +0000 schalla 7059@http://www.datastax.com/support-forums/ There is no schema disagreement before creating a Keyspace. See the output below. I am trying to figure out what is happening here. Connected to: "LabCluster" on 192.168.2.81/9160 Welcome to Cassandra CLI version 1.1.5 Type 'help;' or '?' for help. Type 'quit;' or 'exit;' to quit. [default@unknown] describe cluster; Cluster Information: Snitch: com.datastax.bdp.snitch.DseDelegateSnitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema versions: 67029697-1fcc-3581-9fa1-beb3f516d037: [192.168.2.81, 192.168.2.92, 192.168.2.93] [default@unknown] create keyspace test123; 93c0e215-9327-3998-b3e6-1700fed3cdbf Waiting for schema agreement... The schema has not settled in 10 seconds; further migrations are ill-advised until it does. Versions are 3db42f8b-7749-37cb-80fb-88020a4ab131:[192.168.2.93], 93c0e215-9327-3998-b3e6-1700fed3cdbf:[192.168.2.81, 192.168.2.92] [cassandra@ulvdtlchd01 dse-2.2]$ bin/cassandra-cli -h 192.168.2.81 -p 9160 Connected to: "LabCluster" on 192.168.2.81/9160 Welcome to Cassandra CLI version 1.1.5 Type 'help;' or '?' for help. Type 'quit;' or 'exit;' to quit. [default@unknown] describe cluster; Cluster Information: Snitch: com.datastax.bdp.snitch.DseDelegateSnitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema versions: 93c0e215-9327-3998-b3e6-1700fed3cdbf: [192.168.2.81, 192.168.2.92] c6d8ecc9-08c9-3920-b6b6-c4eed46d074e: [192.168.2.93] alexliu on "Schema versions disagree" http://www.datastax.com/support-forums/topic/schema-versions-disagree#post-7055 Mon, 22 Oct 2012 16:32:16 +0000 alexliu 7055@http://www.datastax.com/support-forums/ To fix schema disagreements, you need to force the disagreeing nodes to rebuild their schema. Here's how: Open the cassandra-cli and run: 'connect localhost/9160;', then 'describe cluster;'. You'll see something like this: [default@unknown] describe cluster; Cluster Information: Snitch: org.apache.cassandra.locator.SimpleSnitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema versions: 75eece10-bf48-11e0-0000-4d205df954a7: [192.168.1.9, 192.168.1.25] 5a54ebd0-bd90-11e0-0000-9510c23fceff: [192.168.1.27] Note which schemas are in the minority and mark down those IPs -- in the above example, 192.168.1.27. Login to each of those machines and cleaninly stop the Cassandra service/process, typically by running: nodetool disablethrift nodetool disablegossip nodetool drain 'sudo service cassandra stop' or 'kill <pid>'. At the end of this process the commit log directory (/var/lib/cassandra/commitlog) should contain only a single small file. Remove the Schema* and Migration* sstables inside of your system keyspace (/var/lib/cassandra/data/system, if you're using the defaults). After starting Cassandra again, this node will notice the missing information and pull in the correct schema from one of the other nodes. In version 1.0.X and before the schema is applied one mutation at a time. While it is being applied the node may log messages, such as the one below, that a Column Family cannot be found. These messages can be ignored. ERROR [MutationStage:1] 2012-05-18 16:23:15,664 RowMutationVerbHandler.java (line 61) Error in row mutation org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1012 To confirm everything is on the same schema, verify that 'describe cluster;' only returns one schema version. alexliu on "Schema versions disagree" http://www.datastax.com/support-forums/topic/schema-versions-disagree#post-7054 Mon, 22 Oct 2012 16:27:15 +0000 alexliu 7054@http://www.datastax.com/support-forums/ To correct this problem: Using the Cassandra CLI, enter the DESCRIBE CLUSTER command. The output contains the IDs of the schema versions and their corresponding nodes. Remove the problematic schema and migration sstables in your system keyspace as described on the Cassandra wiki. To prevent this problem: Perform schema changes one at a time, at a steady pace, and from the same node. If you can't resolve the issue, please attach the script, we can do more investigation. schalla on "Schema versions disagree" http://www.datastax.com/support-forums/topic/schema-versions-disagree#post-7051 Mon, 22 Oct 2012 14:26:42 +0000 schalla 7051@http://www.datastax.com/support-forums/ I am calling "describe cluster" before and after the script. Before running the script, schema versions are consistent and ffter running the script, it creates Keyspace but exits with schema disagreement. Sven on "Schema versions disagree" http://www.datastax.com/support-forums/topic/schema-versions-disagree#post-7039 Sun, 21 Oct 2012 15:44:05 +0000 Sven 7039@http://www.datastax.com/support-forums/ At what point are you calling "describe cluster" Between creating the keyspace and creating the cf's, or after the script failed? schalla on "Schema versions disagree" http://www.datastax.com/support-forums/topic/schema-versions-disagree#post-7024 Fri, 19 Oct 2012 18:03:57 +0000 schalla 7024@http://www.datastax.com/support-forums/ I am using DSE-2.2 and trying to create few keyspaces and CF's in a 3 node cluster. I am using cassandra-cli and passing a script which creates Keyspace and CF's. Script exits right after creating Keyspace and before creating CF's throwing the below. Schema versions disagree, (try again later). However, "describe cluster;" shows the same schema across the cluster. What could be going wrong?