lilyevsky Software, Support, and Training for Apache Cassandra en-US Wed, 19 Jun 2013 07:52:23 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/cant-start-cassandra-node#post-8317 Mon, 07 Jan 2013 23:27:20 +0000 umermk3 8317@http://www.datastax.com/support-forums/ <p>Warning: This will *delete* your old Cassandra data.</p> <p>On OSX or Linux, delete the previous Cassandra folder like this:</p> <p><code>$ sudo rm -fr /var/lib/cassandra</code></p> <p>Depending on your permission, you might not need to use <code>sudo</code>.</p> <p>It is because the new default partitioner is now Murm3r, instead of RandomPartitioner. That's how it worked for me. </p> http://www.datastax.com/support-forums/topic/cant-start-cassandra-node#post-6590 Thu, 20 Sep 2012 08:49:42 +0000 Piotr Kołaczkowski 6590@http://www.datastax.com/support-forums/ <p>Unfortunately bugs happen in all the software - even the most tested, mature and production ready. This is just the real world. However, I can assure you, the support team never leaves any customer with unresolved problem, whatever it is, a real bug, a misconfiguration or misunderstanding how things should work. </p> <p>To avoid bugs like that we ship DSE with selected, stable versions of Cassandra and also give it a good portion of our own systematic testing. </p> <p>If you need to run Cassandra in production, I recommend getting DSE with support subscription - then you get access to a different support system where you get guaranteed 24/7 support with very short reaction times. So, no, you don't need to be an expert to successfully operate it, but knowing some internals really does help, and xedin wanted to say just and only that. We are very sorry if you felt offended. :) </p> http://www.datastax.com/support-forums/topic/cant-start-cassandra-node#post-6588 Thu, 20 Sep 2012 03:49:33 +0000 sroberts 6588@http://www.datastax.com/support-forums/ <p>For anyone who finds this thread, it appears that it was a bug after all.</p> <p><a href="http://www.datastax.com/support-forums/topic/cassandra-does-not-startup" rel="nofollow">http://www.datastax.com/support-forums/topic/cassandra-does-not-startup</a> </p> http://www.datastax.com/support-forums/topic/cant-start-cassandra-node#post-6587 Thu, 20 Sep 2012 03:45:08 +0000 sroberts 6587@http://www.datastax.com/support-forums/ <p>Wow, is xedin a DataStax employee? What an unpleasant disposition. I'm hesitant to pony up nearly $50K/year if that's the kind of attitude I'm going to get from support.</p> <p>Dear xedin, sometimes people inherit tech from former employees and are struggling to get up to speed and keep the systems running. I had this exact error after restarting one of the nodes. The other 9 nodes all restarted just fine, but one of them keeps giving this error. All I did is restart the node (shutdown -r now) and then try to start cassandra. Do I really need to know the organization of all of the files and file systems just to restart a node? Is it too much to ask for an "enterprise class" data storage system to survive a reboot?</p> <p>If we were cassandra experts then we wouldn't need these forums ... and we wouldn't need to pay DataStax insane amounts of money to help us out. </p> http://www.datastax.com/support-forums/topic/cannot-create-cf-in-cassandra-112#post-4447 Tue, 31 Jul 2012 17:57:56 +0000 lilyevsky 4447@http://www.datastax.com/support-forums/ <p>My cassandra used to work fine, now it does not create a table. No error message, but the table is not there. I execute "create table" statement many times, the same thing. When I do it, I see in the system.log the the lines below.<br /> What can be the problem?<br /> Pre-history: before I noticed it is broken, I went through some issues related to the lack of space, so I was adding more storage and restarting both servers (I have my thing partitioned between 2 nodes).<br /> Is there any workaround to bring it back to normal, besides cleaning it all up abd starting from scratch? </p> <p>=====================================================</p> <p> INFO [MigrationStage:1] 2012-07-31 13:51:20,090 ColumnFamilyStore.java (line 643) Enqueuing flush of Memtable-schema_columnfamilies@766316124(1520/1900 serialized/live bytes, 20 ops)<br /> INFO [FlushWriter:5] 2012-07-31 13:51:20,091 Memtable.java (line 266) Writing Memtable-schema_columnfamilies@766316124(1520/1900 serialized/live bytes, 20 ops)<br /> INFO [FlushWriter:5] 2012-07-31 13:51:20,155 Memtable.java (line 307) Completed flushing /Data_Store1/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hd-54-Data.db (1575 bytes) for commitlog position ReplayPosition(segmentId=3346642617210212, position=12976)<br /> INFO [MigrationStage:1] 2012-07-31 13:51:20,156 ColumnFamilyStore.java (line 643) Enqueuing flush of Memtable-schema_columns@2106973490(2048/2560 serialized/live bytes, 30 ops)<br /> INFO [FlushWriter:5] 2012-07-31 13:51:20,157 Memtable.java (line 266) Writing Memtable-schema_columns@2106973490(2048/2560 serialized/live bytes, 30 ops)<br /> INFO [FlushWriter:5] 2012-07-31 13:51:20,211 Memtable.java (line 307) Completed flushing /Data_Store1/cassandra/data/system/schema_columns/system-schema_columns-hd-54-Data.db (2111 bytes) for commitlog position ReplayPosition(segmentId=3346642617210212, position=12976) </p> http://www.datastax.com/support-forums/topic/cant-start-cassandra-node#post-2448 Thu, 28 Jun 2012 16:06:37 +0000 xedin 2448@http://www.datastax.com/support-forums/ <p>Like any other system Cassandra depends on the people how operate it, when you understand how and why system operates in different situations gives you a feel of associated limitations and tradeoffs especially this concerns distributed systems. That I'm trying to say is - when node goes down and couldn't start back up again there are different set of circumstances to investigate and one of them is how the system was operated.</p> <p>This should be of help to understand schema <a href="http://www.datastax.com/dev/blog/the-schema-management-renaissance" rel="nofollow">http://www.datastax.com/dev/blog/the-schema-management-renaissance</a> </p> http://www.datastax.com/support-forums/topic/cant-start-cassandra-node#post-2447 Thu, 28 Jun 2012 15:52:33 +0000 lilyevsky 2447@http://www.datastax.com/support-forums/ <p>It is true that I am very new to Cassandra, and I don't know much about its internals.<br /> On the other hand, I believe that I know what I am doing, because for all my operations I only use cqlsh and Java API. In Java, I only use the CQL part: I prepare the statements and then I execute them.<br /> Is it possible that the second node brings a problem? For example, I create a table, then I drop it, then create again. In the mean time, the table definition went to the second node. Also, during this process, my first node died (NullPointerException - this is a separate issue), so I restarted it at some point. Could it get conflicting table definitions from cqlsh and the other node at the same time?</p> <p>I am not saying that Cassandra 1.1.1 is not ready because I saw an exception. I was not just an exception: the server could not start at all. And I did not manually touch any of Cassandra files, I only used the provided tools. In such case, it does not matter what commands I issue to those tools, it cannot be qualified as "user bad practice". </p> http://www.datastax.com/support-forums/topic/cant-start-cassandra-node#post-2446 Thu, 28 Jun 2012 15:23:07 +0000 xedin 2446@http://www.datastax.com/support-forums/ <p>How do you see the answer for that question then? Would it be it if I add that non of the 1.1.1 adopters reported the same/similar error in Cassandra JIRA yet?</p> <p>When I wrote "delete sstables for column family system.schema_columnfamilies" I meant *all* of the files in the system/schema_columnfamilies directory. </p> <p>Your first paragraph makes it pretty clear that you have no idea about Cassandra's file organization which is very important part of being able to operate the database (understand limitations, tune parameters, choose compaction strategy etc.) so I can conclude that you or ailinykh did something that lead to this kind of error out of not knowing exactly what you were doing.</p> <p>If you or ailinykh could provide any information about the size of the cluster and actions you did with schema that would be helpful to analyze the situation which would result in bug report in C* or user bad practice report.</p> <p>The point "ok here is the exception I got after restarting the node that is why db is not ready for production" is just counter-productive . </p> http://www.datastax.com/support-forums/topic/cant-start-cassandra-node#post-2445 Thu, 28 Jun 2012 14:53:33 +0000 lilyevsky 2445@http://www.datastax.com/support-forums/ <p>I have the same problem. What exactly should I delete? I see many files under schema_columnfamilies, looks like it is 5 files for each family. I guess, I should delete the 5 files belonging to the problematic family, but how do I know which one?</p> <p>Now, about the second question asked by ailinykh, whether it is ready for production. The answer was "I think that it's pretty stable release yes". This is not an answer to the question.</p> <p>Maybe it is "pretty stable", but with error like this, I don't think it can go to production yet. </p> http://www.datastax.com/support-forums/topic/cant-start-cassandra-node#post-2422 Wed, 27 Jun 2012 10:07:45 +0000 xedin 2422@http://www.datastax.com/support-forums/ <p>This shows up that you had two column families with the same id... The easiest way probably would be to just delete sstables for column family system.schema_columnfamilies and after nodes restart - re-create schema using CLI or CQL on one of the nodes. I think that it's pretty stable release yes. </p> http://www.datastax.com/support-forums/topic/cant-start-cassandra-node#post-2416 Wed, 27 Jun 2012 01:22:30 +0000 Anonymous 2416@http://www.datastax.com/support-forums/ <p>Hello, everybody!<br /> I'm running cassandra 1.1.1. Today I decided to restart nodes. But unfortunately, I can't. Every time I get this error message.<br /> Any idea how to fix it? And one more question is 1.1.1 solid enough to run it in production?</p> <p>Thank you,<br /> Andrey</p> <p>INFO 01:08:48,871 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hd-2 (486 bytes)<br /> INFO 01:08:48,874 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-hd-3 (486 bytes)<br /> INFO 01:08:48,880 Opening /var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hd-5 (461 bytes)<br /> ERROR 01:08:49,244 Exception encountered during startup<br /> java.lang.IllegalArgumentException: value already present: 1013<br /> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)<br /> at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)<br /> at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)<br /> at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)<br /> at org.apache.cassandra.config.Schema.load(Schema.java:385)<br /> at org.apache.cassandra.config.Schema.load(Schema.java:106)<br /> at org.apache.cassandra.config.Schema.load(Schema.java:91)<br /> at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:535)<br /> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:182)<br /> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353)<br /> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106)<br /> java.lang.IllegalArgumentException: value already present: 1013<br /> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)<br /> at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)<br /> at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)<br /> at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)<br /> at org.apache.cassandra.config.Schema.load(Schema.java:385)<br /> at org.apache.cassandra.config.Schema.load(Schema.java:106)<br /> at org.apache.cassandra.config.Schema.load(Schema.java:91)<br /> at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:535)<br /> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:182)<br /> at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353)<br /> at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106)<br /> Exception encountered during startup: value already present: 1013 </p>