http://www.datastax.com/support-forums/topic/unable-to-startup-cassandra-nodes-after-unexpected-cluster-shutdown Software, Support, and Training for Apache Cassandra en-US Thu, 23 May 2013 22:22:57 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/unable-to-startup-cassandra-nodes-after-unexpected-cluster-shutdown#post-2027 Mon, 04 Jun 2012 08:02:29 +0000 Anonymous 2027@http://www.datastax.com/support-forums/ <p>Yes. You should install solr libraries on all nodes of the cluster, even on those not using solr search directly. This is because some solr related config is kept within the solr indexed columns and probably when you restart your cluster, the code for checking and synchronizing database schema needs to understand this on all nodes. However, to be sure why is that really needed, I have to ask someone who knows dse search code base better. </p> http://www.datastax.com/support-forums/topic/unable-to-startup-cassandra-nodes-after-unexpected-cluster-shutdown#post-2026 Sun, 03 Jun 2012 21:53:32 +0000 Anonymous 2026@http://www.datastax.com/support-forums/ <p>tjake, Could that happens because the dse-libsolr is not installed on Cassandra nodes? </p> http://www.datastax.com/support-forums/topic/unable-to-startup-cassandra-nodes-after-unexpected-cluster-shutdown#post-2018 Thu, 31 May 2012 15:48:13 +0000 Anonymous 2018@http://www.datastax.com/support-forums/ <p>All nodes are using DSE 2.0. The only deleting of keyspace data folder helps and then all working as expected until next hard reset shutdown... </p> http://www.datastax.com/support-forums/topic/unable-to-startup-cassandra-nodes-after-unexpected-cluster-shutdown#post-2017 Thu, 31 May 2012 15:20:27 +0000 tjake 2017@http://www.datastax.com/support-forums/ <p>Are all nodes using DSE? It seems to me that you have some nodes that are purely Cassandra and can't find the solr jars in the classpath. </p> http://www.datastax.com/support-forums/topic/unable-to-startup-cassandra-nodes-after-unexpected-cluster-shutdown#post-2005 Tue, 29 May 2012 11:59:59 +0000 Anonymous 2005@http://www.datastax.com/support-forums/ <p>On my virtual environment I configured dse cluster with 3 Cassandra nodes and 2 Solr nodes. This is my ring info:</p> <blockquote><p> Address DC Rack Workload Status State Load Owns Token<br /> 113427455640312821154458202477256070485<br /> 192.168.1.61 Cassandra rack1 Cassandra Up Normal 234.94 KB 33.33% 0<br /> 192.168.1.64 Solr rack1 Search Up Normal 234.94 KB 0.00% 20<br /> 192.168.1.62 Cassandra rack1 Cassandra Up Normal 234.94 KB 33.33% 56713727820156410577229101238628035242<br /> 192.168.1.65 Solr rack1 Search Up Normal 242.31 KB 16.67% 85070591730234615865843651857942052864<br /> 192.168.1.63 Cassandra rack1 Cassandra Up Normal 234.94 KB 16.67% 113427455640312821154458202477256070485 </p></blockquote> <p>After unexpected cluster shutdown all nodes are unable to startup. By deleting index <strong>write.lock</strong> file on Solr nodes bring them alive but 3 Cassandra nodes still cannot startup. Below is error entry from output.log.</p> <blockquote><p> INFO 14:42:32,218 Creating new index : ColumnDefinition{name=5f646f63426f6f7374, validator=org.apache.cassandra.db.marshal.UTF8Type, index_type=CUSTOM, index_name='TestNamespace_TestIndex__docBoost_index'}<br /> java.lang.reflect.InvocationTargetException<br /> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)<br /> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)<br /> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br /> at java.lang.reflect.Method.invoke(Method.java:616)<br /> at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)<br /> Caused by: java.lang.NoClassDefFoundError: org/apache/solr/common/params/SolrParams<br /> at java.lang.Class.forName0(Native Method)<br /> at java.lang.Class.forName(Class.java:186)<br /> at org.apache.cassandra.db.index.SecondaryIndex.createInstance(SecondaryIndex.java:307)<br /> at org.apache.cassandra.db.index.SecondaryIndexManager.addIndexedColumn(SecondaryIndexManager.java:188)<br /> at org.apache.cassandra.db.ColumnFamilyStore.&lt;init&gt;(ColumnFamilyStore.java:254)<br /> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:347)<br /> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:319)<br /> at org.apache.cassandra.db.Table.initCf(Table.java:369)<br /> at org.apache.cassandra.db.Table.&lt;init&gt;(Table.java:318)<br /> at org.apache.cassandra.db.Table.open(Table.java:119)<br /> at org.apache.cassandra.db.Table.open(Table.java:102)<br /> at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:202)<br /> at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:105)<br /> at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:257)<br /> ... 5 more<br /> Caused by: java.lang.ClassNotFoundException: org.apache.solr.common.params.SolrParams<br /> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)<br /> at java.security.AccessController.doPrivileged(Native Method)<br /> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)<br /> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)<br /> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)<br /> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)<br /> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:334)<br /> ... 19 more</p> <p>Cannot load daemon<br /> Service exit with a return value of 3 </p></blockquote> <p>Does anybody know how to bring them alive without loosing of data stored on them? </p>