Support Forums : DataStax

DataStax Support Forums » OpsCenter

OpsCenter cannot detect the newly added node

(10 posts) (4 voices)

Started 7 months ago by GodenYao
Latest reply from tupton
This topic is resolved

Tags:

No tags yet.

GodenYao
Member

I had 4 node cassandra cluster running well, all agents connected.
I'm adding 5th node.
nodetools can detect it and it's joining the cluster (about 2-3TB data to transfer). nodetools shows its status is "joining".

I went back to OpsCenter -> Edit Cluster: I added the 5th node's IP address.
manually package installed agent on 5th node.
however, i cannot see OpsCenter find this node.
no errors in OpsCenter log -

here's the 5th node local agent log:
============================================
Starting OpsCenter agent monitor opscenter_agent_monitor.
Starting OpsCenter agent monitor opscenter_agent_monitor.
INFO [main] 2012-10-22 11:28:21,900 Loading conf files: /var/lib/opscenter-agent/conf/address.yaml
INFO [main] 2012-10-22 11:28:21,964 Java vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_09
INFO [main] 2012-10-22 11:28:21,965 Waiting for the config from OpsCenter
INFO [main] 2012-10-22 11:28:21,965 SSL communication is enabled
INFO [main] 2012-10-22 11:28:21,966 Creating stomp connection to 10.2.255.7:61620
INFO [StompConnection receiver] 2012-10-22 11:28:21,973 Reconnecting in 0s.
INFO [main] 2012-10-22 11:28:21,974 Attempting to determine Cassandra's broadcast address through JMX
INFO [Initialization] 2012-10-22 11:28:21,982 Sleeping for 2s before trying to determine IP over JMX again
WARN [pool-2-thread-1] 2012-10-22 11:28:21,998 Tried to send message while not connected: /conf-request [null,["2.10","\/1921359053\/conf",["fe80:0:0:0:2e41:38ff:fe4f:7cd7%2","10.2.255.7","0:0:0:0:0:0:0:1%1","127.0.0.1"]]]
ERROR [StompConnection receiver] 2012-10-22 11:28:22,166 failed connecting to 10.2.255.7:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2012-10-22 11:28:22,183 Reconnecting in 2s.
INFO [Initialization] 2012-10-22 11:28:23,984 Sleeping for 4s before trying to determine IP over JMX again
WARN [pool-2-thread-1] 2012-10-22 11:28:23,986 Tried to send message while not connected: /conf-request [null,["2.10","\/1921359053\/conf",["fe80:0:0:0:2e41:38ff:fe4f:7cd7%2","10.2.255.7","0:0:0:0:0:0:0:1%1","127.0.0.1"]]]
ERROR [StompConnection receiver] 2012-10-22 11:28:24,185 failed connecting to 10.2.255.7:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2012-10-22 11:28:24,185 Reconnecting in 6s.
INFO [Initialization] 2012-10-22 11:28:27,986 Sleeping for 8s before trying to determine IP over JMX again
WARN [pool-2-thread-1] 2012-10-22 11:28:27,987 Tried to send message while not connected: /conf-request [null,["2.10","\/1921359053\/conf",["fe80:0:0:0:2e41:38ff:fe4f:7cd7%2","10.2.255.7","0:0:0:0:0:0:0:1%1","127.0.0.1"]]]
ERROR [StompConnection receiver] 2012-10-22 11:28:30,187 failed connecting to 10.2.255.7:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2012-10-22 11:28:30,187 Reconnecting in 14s.
INFO [main] 2012-10-22 11:28:31,490 Loading conf files: ./conf/address.yaml
INFO [Initialization] 2012-10-22 11:28:35,988 Sleeping for 16s before trying to determine IP over JMX again
WARN [pool-2-thread-1] 2012-10-22 11:28:35,989 Tried to send message while not connected: /conf-request [null,["2.10","\/1921359053\/conf",["fe80:0:0:0:2e41:38ff:fe4f:7cd7%2","10.2.255.7","0:0:0:0:0:0:0:1%1","127.0.0.1"]]]
ERROR [StompConnection receiver] 2012-10-22 11:28:44,191 failed connecting to 10.2.255.7:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2012-10-22 11:28:44,191 Reconnecting in 30s.
INFO [Initialization] 2012-10-22 11:28:51,991 Sleeping for 20s before trying to determine IP over JMX again
WARN [pool-2-thread-1] 2012-10-22 11:28:51,993 Tried to send message while not connected: /conf-request [null,["2.10","\/1921359053\/conf",["fe80:0:0:0:2e41:38ff:fe4f:7cd7%2","10.2.255.7","0:0:0:0:0:0:0:1%1","127.0.0.1"]]]
INFO [Initialization] 2012-10-22 11:29:11,994 Sleeping for 20s before trying to determine IP over JMX again
WARN [pool-2-thread-1] 2012-10-22 11:29:11,995 Tried to send message while not connected: /conf-request [null,["2.10","\/1921359053\/conf",["fe80:0:0:0:2e41:38ff:fe4f:7cd7%2","10.2.255.7","0:0:0:0:0:0:0:1%1","127.0.0.1"]]]
ERROR [StompConnection receiver] 2012-10-22 11:29:14,194 failed connecting to 10.2.255.7:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2012-10-22 11:29:14,194 Reconnecting in 62s.

Posted 7 months ago #
nickmbailey
Key Master

That log is indicating the agent can not connect to OpsCenter on the interface '10.2.255.7' and port '61620'. Can you verify that the cassandra node you are adding doesn't have a firewall preventing the agent from connecting?

Additionally, nodes that are bootstrapping won't show up in the UI until they are done bootstrapping. We have a bug to fix this.

Posted 7 months ago #
GodenYao
Member

i'm sure there's no firewall. so maybe that's just that node is still in the process of joining cluster.
I'll post back when that is done and see if that's the cause.

Posted 7 months ago #
GodenYao
Member

I think i found the issue: in OpsCenter (server) node, here's a line of warning
===================================================================================
WARN: Unable to find a matching cluster for [u'fe80:0:0:0:2e41:38ff:fe4f:7cd7%2', u'10.2.255.7', u'0:0:0:0:0:0:0:1%1', u'127.0.0.1']

How to get this new node into the original cluster that all the rest nodes are in?
I only have 1 cluster.

Posted 7 months ago #
GodenYao
Member

btw, my node 1-4 are running cassandra 1.1.5, 5th is running 1.1.6 - does that matter?

Posted 7 months ago #
nickmbailey
Key Master

You should try to make sure all nodes are on the same version, but it shouldn't matter here. Is the node you added still in the 'joining' state or has it fully joined now? It won't show up in opscenter until it finishes joining.

I would expect the log you are seeing until it finishes joining.

Posted 7 months ago #
GodenYao
Member

there're a couple TB data to move. so it'll be a while.

Posted 7 months ago #
GodenYao
Member

I can see 5th node now after it finished data movement.

Posted 6 months ago #
jurdefur
Member

Got exact the same problem here, but we only have 6gb per node and it still does not recognize the new node. It stays on "joinging" and with gossipinfo it stays on "booting". Any idee how long a join can take?

Posted 5 months ago #
tupton
Moderator

jurdefur,

It's possible that bootstrapping that node somehow failed. If this is the case, you can try again to get it to join. You can stop that node, wipe everything, assign a different token (just subtract one from its existing token to make it easy), and start it up again.

Posted 5 months ago #

RSS feed for this topic

Reply

You must log in to post.