resolved
schalla
Member
I am using DSE-2.2 and trying to create few keyspaces and CF's in a 3 node cluster.
I am using cassandra-cli and passing a script which creates Keyspace and CF's. Script exits right after creating Keyspace and before creating CF's throwing the below.
Schema versions disagree, (try again later).
However, "describe cluster;" shows the same schema across the cluster.
What could be going wrong?
At what point are you calling "describe cluster" Between creating the keyspace and creating the cf's, or after the script failed?
alexliu
Member
To correct this problem:
Using the Cassandra CLI, enter the DESCRIBE CLUSTER command. The output contains the IDs of the schema versions and their corresponding nodes. Remove the problematic schema and migration sstables in your system keyspace as described on the Cassandra wiki.
To prevent this problem:
Perform schema changes one at a time, at a steady pace, and from the same node.
If you can't resolve the issue, please attach the script, we can do more investigation.
alexliu
Member
To fix schema disagreements, you need to force the disagreeing nodes to rebuild their schema. Here's how:
Open the cassandra-cli and run: 'connect localhost/9160;', then 'describe cluster;'. You'll see something like this:
[default@unknown] describe cluster;
Cluster Information:
Snitch: org.apache.cassandra.locator.SimpleSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
75eece10-bf48-11e0-0000-4d205df954a7: [192.168.1.9, 192.168.1.25]
5a54ebd0-bd90-11e0-0000-9510c23fceff: [192.168.1.27]
Note which schemas are in the minority and mark down those IPs -- in the above example, 192.168.1.27. Login to each of those machines and cleaninly stop the Cassandra service/process, typically by running:
nodetool disablethrift
nodetool disablegossip
nodetool drain
'sudo service cassandra stop' or 'kill <pid>'.
At the end of this process the commit log directory (/var/lib/cassandra/commitlog) should contain only a single small file.
Remove the Schema* and Migration* sstables inside of your system keyspace (/var/lib/cassandra/data/system, if you're using the defaults).
After starting Cassandra again, this node will notice the missing information and pull in the correct schema from one of the other nodes. In version 1.0.X and before the schema is applied one mutation at a time. While it is being applied the node may log messages, such as the one below, that a Column Family cannot be found. These messages can be ignored.
ERROR [MutationStage:1] 2012-05-18 16:23:15,664 RowMutationVerbHandler.java (line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1012
To confirm everything is on the same schema, verify that 'describe cluster;' only returns one schema version.
schalla
Member
There is no schema disagreement before creating a Keyspace. See the output below. I am trying to figure out what is happening here.
Connected to: "LabCluster" on 192.168.2.81/9160
Welcome to Cassandra CLI version 1.1.5
Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.
[default@unknown] describe cluster;
Cluster Information:
Snitch: com.datastax.bdp.snitch.DseDelegateSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
67029697-1fcc-3581-9fa1-beb3f516d037: [192.168.2.81, 192.168.2.92, 192.168.2.93]
[default@unknown] create keyspace test123;
93c0e215-9327-3998-b3e6-1700fed3cdbf
Waiting for schema agreement...
The schema has not settled in 10 seconds; further migrations are ill-advised until it does.
Versions are 3db42f8b-7749-37cb-80fb-88020a4ab131:[192.168.2.93], 93c0e215-9327-3998-b3e6-1700fed3cdbf:[192.168.2.81, 192.168.2.92]
[cassandra@ulvdtlchd01 dse-2.2]$ bin/cassandra-cli -h 192.168.2.81 -p 9160
Connected to: "LabCluster" on 192.168.2.81/9160
Welcome to Cassandra CLI version 1.1.5
Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.
[default@unknown] describe cluster;
Cluster Information:
Snitch: com.datastax.bdp.snitch.DseDelegateSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
93c0e215-9327-3998-b3e6-1700fed3cdbf: [192.168.2.81, 192.168.2.92]
c6d8ecc9-08c9-3920-b6b6-c4eed46d074e: [192.168.2.93]
You must log in to post.
