I'm trying to bring up a cluster of 3 cass nodes, 3 analytics nodes and 2 solr nodes. When everything starts up using the DataStax AMI it works ok. If I try to restart any of the analytics nodes using "sudo service dse restart" after changing the mapred-site.xml file in /etc/dse/hadoop it never rejoins the cluster correctly.
Node tool shows all my nodes are analytics nodes?!
ubuntu@ip-10-10-1-161:~$ nodetool ring -h localhost
Address DC Rack Status State Load Owns Token
141784319550391026443072753096570088105
10.10.1.167 Analytics rack1 Down Normal ? 16.67% 0
10.10.1.172 Analytics rack1 Down Normal ? 8.33% 14178431955039102644307275309657008809
10.10.1.161 Analytics rack1 Up Normal 300.96 KB 8.33% 28356863910078205288614550619314017620
10.10.1.165 Analytics rack1 Down Normal ? 16.67% 56713727820156410577229101238628035242
10.10.1.157 Analytics rack1 Down Normal ? 16.67% 85070591730234615865843651857942052862
10.10.1.171 Analytics rack1 Down Normal ? 8.33% 99249023685273718510150927167599061673
10.10.1.163 Analytics rack1 Down Normal ? 8.33% 113427455640312821154458202477256070485
10.10.1.156 Analytics rack1 Down Normal ? 16.67% 141784319550391026443072753096570088105
-------------------------
logfile
I keep seeing this over and over again on the hadoop nodes
INFO 01:35:07,645 problem cleaning system directory: cfs:/raid0/hadoop/cassandra/mapred/system
java.io.IOException: UnavailableException()
at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.readSlice(CassandraFileSystemThriftStore.java:688)
at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.readINodeColumns(CassandraFileSystemThriftStore.java:619)
at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.retrieveINode(CassandraFileSystemThriftStore.java:571)
at com.datastax.bdp.hadoop.cfs.CassandraFileSystemThriftStore.retrieveINode(CassandraFileSystemThriftStore.java:563)
at com.datastax.bdp.hadoop.cfs.CassandraFileSystem.getFileStatus(CassandraFileSystem.java:521)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2379)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2195)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2189)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:303)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
at org.apache.hadoop.mapred.HadoopTrackerPlugin$1.run(HadoopTrackerPlugin.java:237)
at java.lang.Thread.run(Thread.java:662)
