Can't start cassandra node

(10 posts) (6 voices)

• Started 11 months ago by
• Latest reply from umermk3
• This topic is not resolved

Tags:

No tags yet.

1. 

   Anonymous
   Unregistered

   Hello, everybody!
   I'm running cassandra 1.1.1. Today I decided to restart nodes. But unfortunately, I can't. Every time I get this error message.
   Any idea how to fix it? And one more question is 1.1.1 solid enough to run it in production?

   Thank you,
   Andrey

   INFO 01:08:48,871 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hd-2 (486 bytes)
   INFO 01:08:48,874 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hd-3 (486 bytes)
   INFO 01:08:48,880 Opening /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hd-5 (461 bytes)
   ERROR 01:08:49,244 Exception encountered during startup
   java.lang.IllegalArgumentException: value already present: 1013
   at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
   at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
   at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
   at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
   at org.apache.cassandra.config.Schema.load(Schema.java:385)
   at org.apache.cassandra.config.Schema.load(Schema.java:106)
   at org.apache.cassandra.config.Schema.load(Schema.java:91)
   at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:535)
   at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:182)
   at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353)
   at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106)
   java.lang.IllegalArgumentException: value already present: 1013
   at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
   at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
   at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
   at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
   at org.apache.cassandra.config.Schema.load(Schema.java:385)
   at org.apache.cassandra.config.Schema.load(Schema.java:106)
   at org.apache.cassandra.config.Schema.load(Schema.java:91)
   at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:535)
   at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:182)
   at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353)
   at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106)
   Exception encountered during startup: value already present: 1013

   Posted 11 months ago #
2. 

   xedin
   Member

   This shows up that you had two column families with the same id... The easiest way probably would be to just delete sstables for column family system.schema_columnfamilies and after nodes restart - re-create schema using CLI or CQL on one of the nodes. I think that it's pretty stable release yes.

   Posted 11 months ago #
3. 

   lilyevsky
   Member

   I have the same problem. What exactly should I delete? I see many files under schema_columnfamilies, looks like it is 5 files for each family. I guess, I should delete the 5 files belonging to the problematic family, but how do I know which one?

   Now, about the second question asked by ailinykh, whether it is ready for production. The answer was "I think that it's pretty stable release yes". This is not an answer to the question.

   Maybe it is "pretty stable", but with error like this, I don't think it can go to production yet.

   Posted 11 months ago #
4. 

   xedin
   Member

   How do you see the answer for that question then? Would it be it if I add that non of the 1.1.1 adopters reported the same/similar error in Cassandra JIRA yet?

   When I wrote "delete sstables for column family system.schema_columnfamilies" I meant *all* of the files in the system/schema_columnfamilies directory.

   Your first paragraph makes it pretty clear that you have no idea about Cassandra's file organization which is very important part of being able to operate the database (understand limitations, tune parameters, choose compaction strategy etc.) so I can conclude that you or ailinykh did something that lead to this kind of error out of not knowing exactly what you were doing.

   If you or ailinykh could provide any information about the size of the cluster and actions you did with schema that would be helpful to analyze the situation which would result in bug report in C* or user bad practice report.

   The point "ok here is the exception I got after restarting the node that is why db is not ready for production" is just counter-productive .

   Posted 11 months ago #
5. 

   lilyevsky
   Member

   It is true that I am very new to Cassandra, and I don't know much about its internals.
   On the other hand, I believe that I know what I am doing, because for all my operations I only use cqlsh and Java API. In Java, I only use the CQL part: I prepare the statements and then I execute them.
   Is it possible that the second node brings a problem? For example, I create a table, then I drop it, then create again. In the mean time, the table definition went to the second node. Also, during this process, my first node died (NullPointerException - this is a separate issue), so I restarted it at some point. Could it get conflicting table definitions from cqlsh and the other node at the same time?

   I am not saying that Cassandra 1.1.1 is not ready because I saw an exception. I was not just an exception: the server could not start at all. And I did not manually touch any of Cassandra files, I only used the provided tools. In such case, it does not matter what commands I issue to those tools, it cannot be qualified as "user bad practice".

   Posted 11 months ago #
6. 

   xedin
   Member

   Like any other system Cassandra depends on the people how operate it, when you understand how and why system operates in different situations gives you a feel of associated limitations and tradeoffs especially this concerns distributed systems. That I'm trying to say is - when node goes down and couldn't start back up again there are different set of circumstances to investigate and one of them is how the system was operated.

   This should be of help to understand schema http://www.datastax.com/dev/blog/the-schema-management-renaissance

   Posted 11 months ago #
7. 

   sroberts
   Member

   Wow, is xedin a DataStax employee? What an unpleasant disposition. I'm hesitant to pony up nearly $50K/year if that's the kind of attitude I'm going to get from support.

   Dear xedin, sometimes people inherit tech from former employees and are struggling to get up to speed and keep the systems running. I had this exact error after restarting one of the nodes. The other 9 nodes all restarted just fine, but one of them keeps giving this error. All I did is restart the node (shutdown -r now) and then try to start cassandra. Do I really need to know the organization of all of the files and file systems just to restart a node? Is it too much to ask for an "enterprise class" data storage system to survive a reboot?

   If we were cassandra experts then we wouldn't need these forums ... and we wouldn't need to pay DataStax insane amounts of money to help us out.

   Posted 8 months ago #
8. 

   sroberts
   Member

   For anyone who finds this thread, it appears that it was a bug after all.

   http://www.datastax.com/support-forums/topic/cassandra-does-not-startup

   Posted 8 months ago #
9. 

   Piotr Kołaczkowski
   Administrator

   Unfortunately bugs happen in all the software - even the most tested, mature and production ready. This is just the real world. However, I can assure you, the support team never leaves any customer with unresolved problem, whatever it is, a real bug, a misconfiguration or misunderstanding how things should work.

   To avoid bugs like that we ship DSE with selected, stable versions of Cassandra and also give it a good portion of our own systematic testing.

   If you need to run Cassandra in production, I recommend getting DSE with support subscription - then you get access to a different support system where you get guaranteed 24/7 support with very short reaction times. So, no, you don't need to be an expert to successfully operate it, but knowing some internals really does help, and xedin wanted to say just and only that. We are very sorry if you felt offended. :)

   Posted 8 months ago #
10. 

    umermk3
    Member

    Warning: This will *delete* your old Cassandra data.

    On OSX or Linux, delete the previous Cassandra folder like this:

    $ sudo rm -fr /var/lib/cassandra

    Depending on your permission, you might not need to use sudo.

    It is because the new default partitioner is now Murm3r, instead of RandomPartitioner. That's how it worked for me.

    Posted 4 months ago #

RSS feed for this topic