So, we where testing out to nuke one of our nodes in the cluster today (running 1.1.6) and upgrade that node to a bigger machine. When upgrade we noticed that we now got 1.1.7 installed, and that caused some issues.
The entire cluster stopped working, and we got these kinds of errors on the new node:
ERROR [MutationStage:22] 2012-12-21 09:41:30,132 RowMutationVerbHandler.java (line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1011
at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
When I killed the new node, downgraded to 1.1.6, then started it up again. No issues was found and everything was working again. Just a heads up.
Using the RPM packages provided. apache-cassandra11-1.1.6 vs apache-cassandra11-1.1.7