http://www.datastax.com/support-forums/topic/opsc-reporting-corrupt-endpoint-range-after-nodetool-removetoken Software, Support, and Training for Apache Cassandra en-US Sat, 25 May 2013 02:38:11 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/opsc-reporting-corrupt-endpoint-range-after-nodetool-removetoken#post-1311 Tue, 06 Mar 2012 19:26:25 +0000 Anonymous 1311@http://www.datastax.com/support-forums/ <p>Everything else in OpsCenter appeared to be working correctly outside of monitoring the corrupt endpoint range node, which was showing in list view as something other than active-normal (As I said I have brought the downed node back up and into the ring and resolved the issue, but if I had to bet I'd say it was displaying as Unresponsive). This was the only keyspace that was sending the corrupt endpoint token range error in the logs.</p> <p>The seed list in opscenterd.conf contains one node in each DC (And in DC2, it's the node that was still in the ring) </p> http://www.datastax.com/support-forums/topic/opsc-reporting-corrupt-endpoint-range-after-nodetool-removetoken#post-1307 Tue, 06 Mar 2012 18:04:41 +0000 nickmbailey 1307@http://www.datastax.com/support-forums/ <p>This appears to be a bug in how OpsCenter processes the thrift 'describe_ring' call. You are using removetoken correctly. Is OpsCenter still working correctly when you see this? It should be able to process the describe_ring call for a different keyspace correctly and still work as normal.</p> <p>Also is OpsCenter configured to connect to the nodes in the datacenter with 1 node or the datacenter with 2 nodes? I'm referring to the list of seed nodes configured in opscenterd.conf </p> http://www.datastax.com/support-forums/topic/opsc-reporting-corrupt-endpoint-range-after-nodetool-removetoken#post-1303 Tue, 06 Mar 2012 13:06:58 +0000 Anonymous 1303@http://www.datastax.com/support-forums/ <p>I'm testing out Cassandra with a 4 node cluster spread across 2 DCs and on my stress keyspace, Keyspace1, the RF is DC1:1,DC2:1<br /> As part of my testing I had a VM, ip.83, wiped after several stress write runs to see how to handle a node failure. Everything worked as expected, and since I wasn't going to bring back the node for a few days I figured I'd remove its token from the ring with a nodetool removetoken. After the streaming across the remaining replica brought the other node in the same DC, ip.82, up to date with its new ownership load, OpsCenter reported that ip.82 was no longer connected, despite nodetool ring showing it as up. I took a look in the logs and noticed that in the same timestamp that confirmed the nodetool removetoken, I started receiving these errors as well.</p> <p>2012-03-05 08:09:55-0500 [] ERROR: Corrupt endpoint range data for node 10.198.30.82 on keyspace Keyspace1: [TokenRange(end_token='1', start_token='0', endpoints=['10.198.30.82', '10.198.30.81']), TokenRange(end_token='85070591730234615865843651857942052864', start_token='1', endpoints=['10.198.30.82', '10.198.30.81']), TokenRange(end_token='0', start_token='85070591730234615865843651857942052864', endpoints=['10.198.30.82', '10.198.30.80'])]</p> <p>It makes sense for it to now have ownership over the entire token range as it was the only node operating in its DC, but is it intended for some reason I'm just not thinking of for OpsCenter to treat 0-1, 1-85070591730234615865843651857942052864, 85070591730234615865843651857942052864-0 as 3 seperate keyranges, rather than 0-0? (Or in this case 1-1 I suppose, since that is its token) The problem seemed to be confined to OpsCenter as the cluster was still working, albeit expectedly slower with a 25% reduction in nodes.</p> <p>I have since brought back up the downed node and put it in the ring with its previous token and everything is now working again, I was just hoping for some clarification into why this occurred. Is it simply that removetoken shouldn't really be used except for cases of replacing dead nodes? </p>