http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node Software, Support, and Training for Apache Cassandra en-US Wed, 19 Jun 2013 00:47:26 +0000 http://bbpress.org/?v=1.0.3 q http://www.datastax.com/support-forums/search.php http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node#post-7969 Thu, 13 Dec 2012 20:22:19 +0000 tupton 7969@http://www.datastax.com/support-forums/ <p>jurdefur,</p> <p>It's possible that bootstrapping that node somehow failed. If this is the case, you can try again to get it to join. You can stop that node, wipe everything, assign a different token (just subtract one from its existing token to make it easy), and start it up again. </p> http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node#post-7966 Thu, 13 Dec 2012 11:24:46 +0000 jurdefur 7966@http://www.datastax.com/support-forums/ <p>Got exact the same problem here, but we only have 6gb per node and it still does not recognize the new node. It stays on "joinging" and with gossipinfo it stays on "booting". Any idee how long a join can take? </p> http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node#post-7099 Wed, 24 Oct 2012 15:43:07 +0000 GodenYao 7099@http://www.datastax.com/support-forums/ <p>I can see 5th node now after it finished data movement. </p> http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node#post-7089 Tue, 23 Oct 2012 21:59:04 +0000 GodenYao 7089@http://www.datastax.com/support-forums/ <p>there're a couple TB data to move. so it'll be a while. </p> http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node#post-7085 Tue, 23 Oct 2012 15:02:45 +0000 nickmbailey 7085@http://www.datastax.com/support-forums/ <p>You should try to make sure all nodes are on the same version, but it shouldn't matter here. Is the node you added still in the 'joining' state or has it fully joined now? It won't show up in opscenter until it finishes joining.</p> <p>I would expect the log you are seeing until it finishes joining. </p> http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node#post-7067 Tue, 23 Oct 2012 05:38:16 +0000 GodenYao 7067@http://www.datastax.com/support-forums/ <p>btw, my node 1-4 are running cassandra 1.1.5, 5th is running 1.1.6 - does that matter? </p> http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node#post-7066 Tue, 23 Oct 2012 04:43:57 +0000 GodenYao 7066@http://www.datastax.com/support-forums/ <p>I think i found the issue: in OpsCenter (server) node, here's a line of warning<br /> ===================================================================================<br /> WARN: Unable to find a matching cluster for [u'fe80:0:0:0:2e41:38ff:fe4f:7cd7%2', u'10.2.255.7', u'0:0:0:0:0:0:0:1%1', u'127.0.0.1']</p> <p>How to get this new node into the original cluster that all the rest nodes are in?<br /> I only have 1 cluster. </p> http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node#post-7064 Mon, 22 Oct 2012 21:14:22 +0000 GodenYao 7064@http://www.datastax.com/support-forums/ <p>i'm sure there's no firewall. so maybe that's just that node is still in the process of joining cluster.<br /> I'll post back when that is done and see if that's the cause. </p> http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node#post-7063 Mon, 22 Oct 2012 20:52:58 +0000 nickmbailey 7063@http://www.datastax.com/support-forums/ <p>That log is indicating the agent can not connect to OpsCenter on the interface '10.2.255.7' and port '61620'. Can you verify that the cassandra node you are adding doesn't have a firewall preventing the agent from connecting?</p> <p>Additionally, nodes that are bootstrapping won't show up in the UI until they are done bootstrapping. We have a bug to fix this. </p> http://www.datastax.com/support-forums/topic/opscenter-cannot-detect-the-newly-added-node#post-7062 Mon, 22 Oct 2012 20:44:45 +0000 GodenYao 7062@http://www.datastax.com/support-forums/ <p>I had 4 node cassandra cluster running well, all agents connected.<br /> I'm adding 5th node.<br /> nodetools can detect it and it's joining the cluster (about 2-3TB data to transfer). nodetools shows its status is "joining".</p> <p>I went back to OpsCenter -&gt; Edit Cluster: I added the 5th node's IP address.<br /> manually package installed agent on 5th node.<br /> however, i cannot see OpsCenter find this node.<br /> no errors in OpsCenter log - </p> <p>here's the 5th node local agent log:<br /> ============================================<br /> Starting OpsCenter agent monitor opscenter_agent_monitor.<br /> Starting OpsCenter agent monitor opscenter_agent_monitor.<br /> INFO [main] 2012-10-22 11:28:21,900 Loading conf files: /var/lib/opscenter-agent/conf/address.yaml<br /> INFO [main] 2012-10-22 11:28:21,964 Java vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_09<br /> INFO [main] 2012-10-22 11:28:21,965 Waiting for the config from OpsCenter<br /> INFO [main] 2012-10-22 11:28:21,965 SSL communication is enabled<br /> INFO [main] 2012-10-22 11:28:21,966 Creating stomp connection to 10.2.255.7:61620<br /> INFO [StompConnection receiver] 2012-10-22 11:28:21,973 Reconnecting in 0s.<br /> INFO [main] 2012-10-22 11:28:21,974 Attempting to determine Cassandra's broadcast address through JMX<br /> INFO [Initialization] 2012-10-22 11:28:21,982 Sleeping for 2s before trying to determine IP over JMX again<br /> WARN [pool-2-thread-1] 2012-10-22 11:28:21,998 Tried to send message while not connected: /conf-request [null,["2.10","\/1921359053\/conf",["fe80:0:0:0:2e41:38ff:fe4f:7cd7%2","10.2.255.7","0:0:0:0:0:0:0:1%1","127.0.0.1"]]]<br /> ERROR [StompConnection receiver] 2012-10-22 11:28:22,166 failed connecting to 10.2.255.7:61620:java.net.ConnectException: Connection refused<br /> INFO [StompConnection receiver] 2012-10-22 11:28:22,183 Reconnecting in 2s.<br /> INFO [Initialization] 2012-10-22 11:28:23,984 Sleeping for 4s before trying to determine IP over JMX again<br /> WARN [pool-2-thread-1] 2012-10-22 11:28:23,986 Tried to send message while not connected: /conf-request [null,["2.10","\/1921359053\/conf",["fe80:0:0:0:2e41:38ff:fe4f:7cd7%2","10.2.255.7","0:0:0:0:0:0:0:1%1","127.0.0.1"]]]<br /> ERROR [StompConnection receiver] 2012-10-22 11:28:24,185 failed connecting to 10.2.255.7:61620:java.net.ConnectException: Connection refused<br /> INFO [StompConnection receiver] 2012-10-22 11:28:24,185 Reconnecting in 6s.<br /> INFO [Initialization] 2012-10-22 11:28:27,986 Sleeping for 8s before trying to determine IP over JMX again<br /> WARN [pool-2-thread-1] 2012-10-22 11:28:27,987 Tried to send message while not connected: /conf-request [null,["2.10","\/1921359053\/conf",["fe80:0:0:0:2e41:38ff:fe4f:7cd7%2","10.2.255.7","0:0:0:0:0:0:0:1%1","127.0.0.1"]]]<br /> ERROR [StompConnection receiver] 2012-10-22 11:28:30,187 failed connecting to 10.2.255.7:61620:java.net.ConnectException: Connection refused<br /> INFO [StompConnection receiver] 2012-10-22 11:28:30,187 Reconnecting in 14s.<br /> INFO [main] 2012-10-22 11:28:31,490 Loading conf files: ./conf/address.yaml<br /> INFO [Initialization] 2012-10-22 11:28:35,988 Sleeping for 16s before trying to determine IP over JMX again<br /> WARN [pool-2-thread-1] 2012-10-22 11:28:35,989 Tried to send message while not connected: /conf-request [null,["2.10","\/1921359053\/conf",["fe80:0:0:0:2e41:38ff:fe4f:7cd7%2","10.2.255.7","0:0:0:0:0:0:0:1%1","127.0.0.1"]]]<br /> ERROR [StompConnection receiver] 2012-10-22 11:28:44,191 failed connecting to 10.2.255.7:61620:java.net.ConnectException: Connection refused<br /> INFO [StompConnection receiver] 2012-10-22 11:28:44,191 Reconnecting in 30s.<br /> INFO [Initialization] 2012-10-22 11:28:51,991 Sleeping for 20s before trying to determine IP over JMX again<br /> WARN [pool-2-thread-1] 2012-10-22 11:28:51,993 Tried to send message while not connected: /conf-request [null,["2.10","\/1921359053\/conf",["fe80:0:0:0:2e41:38ff:fe4f:7cd7%2","10.2.255.7","0:0:0:0:0:0:0:1%1","127.0.0.1"]]]<br /> INFO [Initialization] 2012-10-22 11:29:11,994 Sleeping for 20s before trying to determine IP over JMX again<br /> WARN [pool-2-thread-1] 2012-10-22 11:29:11,995 Tried to send message while not connected: /conf-request [null,["2.10","\/1921359053\/conf",["fe80:0:0:0:2e41:38ff:fe4f:7cd7%2","10.2.255.7","0:0:0:0:0:0:0:1%1","127.0.0.1"]]]<br /> ERROR [StompConnection receiver] 2012-10-22 11:29:14,194 failed connecting to 10.2.255.7:61620:java.net.ConnectException: Connection refused<br /> INFO [StompConnection receiver] 2012-10-22 11:29:14,194 Reconnecting in 62s. </p>