http://www.datastax.com/support-forums/topic/cannot-get-analytic-nodes-to-re-join-the-cluster Software, Support, and Training for Apache Cassandra en-US Thu, 20 Jun 2013 08:44:58 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/cannot-get-analytic-nodes-to-re-join-the-cluster#post-9195 Wed, 27 Feb 2013 18:51:50 +0000 alexliu 9195@http://www.datastax.com/support-forums/ <p>Can you first start clusters with default settings, then check the ring to make sure all nodes start correctly.</p> <p>Make change to mapred-site.xml for a node and restart it, wait until the ring info is showing the node correctly joining the ring. Do it for each node once a time.</p> <p>If you still have issue, can you post the changes you made to mapred-site.xml? </p> http://www.datastax.com/support-forums/topic/cannot-get-analytic-nodes-to-re-join-the-cluster#post-9065 Tue, 19 Feb 2013 01:42:38 +0000 devnulldude 9065@http://www.datastax.com/support-forums/ <p>I'm trying to bring up a cluster of 3 cass nodes, 3 analytics nodes and 2 solr nodes. When everything starts up using the DataStax AMI it works ok. If I try to restart any of the analytics nodes using "sudo service dse restart" after changing the mapred-site.xml file in /etc/dse/hadoop it never rejoins the cluster correctly. </p> <p>Node tool shows all my nodes are analytics nodes?!<br /> ubuntu@ip-10-10-1-161:~$ nodetool ring -h localhost<br /> Address DC Rack Status State Load Owns Token<br /> 141784319550391026443072753096570088105<br /> 10.10.1.167 Analytics rack1 Down Normal ? 16.67% 0<br /> 10.10.1.172 Analytics rack1 Down Normal ? 8.33% 14178431955039102644307275309657008809<br /> 10.10.1.161 Analytics rack1 Up Normal 300.96 KB 8.33% 28356863910078205288614550619314017620<br /> 10.10.1.165 Analytics rack1 Down Normal ? 16.67% 56713727820156410577229101238628035242<br /> 10.10.1.157 Analytics rack1 Down Normal ? 16.67% 85070591730234615865843651857942052862<br /> 10.10.1.171 Analytics rack1 Down Normal ? 8.33% 99249023685273718510150927167599061673<br /> 10.10.1.163 Analytics rack1 Down Normal ? 8.33% 113427455640312821154458202477256070485<br /> 10.10.1.156 Analytics rack1 Down Normal ? 16.67% 141784319550391026443072753096570088105 </p> <p>-------------------------<br /> logfile </p> <p>I keep seeing this over and over again on the hadoop nodes</p> <p>INFO 01:35:07,645 problem cleaning system directory: cfs:/raid0/hadoop/cassandra/mapred/system<br /> java.io.IOException: UnavailableException()<br /> at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.readSlice(CassandraFileSystemThriftStore.java:688)<br /> at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.readINodeColumns(CassandraFileSystemThriftStore.java:619)<br /> at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.retrieveINode(CassandraFileSystemThriftStore.java:571)<br /> at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.retrieveINode(CassandraFileSystemThriftStore.java:563)<br /> at com.datastax.bdp.hadoop.cfs.CassandraFileSystem.getFileStatus(CassandraFileSystem.java:521)<br /> at org.apache.hadoop.mapred.JobTracker.&lt;init&gt;(JobTracker.java:2379)<br /> at org.apache.hadoop.mapred.JobTracker.&lt;init&gt;(JobTracker.java:2195)<br /> at org.apache.hadoop.mapred.JobTracker.&lt;init&gt;(JobTracker.java:2189)<br /> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:303)<br /> at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)<br /> at org.apache.hadoop.mapred.HadoopTrackerPlugin$1.run(HadoopTrackerPlugin.java:237)<br /> at java.lang.Thread.run(Thread.java:662) </p>