Support Forums : DataStax

DataStax Support Forums » OpsCenter

Agents not connecting to OpsCenter

(25 posts) (3 voices)

Started 11 months ago by swoo
Latest reply from swoo
This topic is not resolved

Tags:

No tags yet.

12 Next »

swoo
Member

We just installed OpsCenter and the OpsCenter-agent. We have 4 agents. 3 of the 4 agents are not connecting to OpsCenter.
I'm getting the following error:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
INFO [StompConnection receiver] 2012-07-09 20:54:10,613 Reconnecting in 0s.
ERROR [StompConnection receiver] 2012-07-09 20:54:10,688 failed connecting to 10.1.18.32:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2012-07-09 20:54:10,707 Reconnecting in 2s.
ERROR [StompConnection receiver] 2012-07-09 20:54:12,708 failed connecting to 10.1.18.32:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2012-07-09 20:54:12,708 Reconnecting in 6s.
ERROR [StompConnection receiver] 2012-07-09 20:54:18,709 failed connecting to 10.1.18.32:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2012-07-09 20:54:18,709 Reconnecting in 14s.
ERROR [StompConnection receiver] 2012-07-09 20:54:32,710 failed connecting to 10.1.18.32:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2012-07-09 20:54:32,710 Reconnecting in 30s.
ERROR [StompConnection receiver] 2012-07-09 20:55:02,711 failed connecting to 10.1.18.32:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2012-07-09 20:55:02,711 Reconnecting in 62s.
ERROR [StompConnection receiver] 2012-07-09 20:56:04,712 failed connecting to 10.1.18.32:61620:java.net.ConnectException: Connection refused

Help on figuring out what's wrong would be appreciated. Thanks, Shirley

Posted 11 months ago #
thobbs
Administrator

First, is it possible that a firewall is blocking port 61620 on the machine running opscenterd?

Second, do you see any errors on the opscenterd log? (It's located at /var/log/opscenterd/opscenterd.log if you installed the deb or rpm package.)

Posted 11 months ago #
swoo
Member

The log has:

2012-07-09 21:06:36+0000 [PE_Cluster] DEBUG: Performing HTTP request: https://10.1.18.32:61621/cluster/rack?node_ip=10.1.18.32, body: {}
2012-07-09 21:06:36+0000 [PE_Cluster] WARN: HTTP request https://10.1.18.32:61621/cluster/rack?node_ip=10.1.18.32 failed: Connection was refused by other side: 111: Connection refused.
2012-07-09 21:06:36+0000 [PE_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to https://10.1.18.32:61621/cluster/rack?node_ip=10.1.18.32 : Connection was refused by other side: 111: Connection refused.

Posted 11 months ago #
swoo
Member

telnet to the port from the agent to the opscenter server works.
iptables is turned off on all machines.
But still cannot connect.

Posted 11 months ago #
nickmbailey
Key Master

Just to verify:

from all agent machines the following works:

telnet 10.1.18.32 61620

And from the opscenter machine the following works:

telnet 10.1.18.32 61621

If that is the case then something else is going wrong. We would likely need a full agent and opscenter log in order to debug.

Posted 11 months ago #
swoo
Member

from all agent machines, telnet 10.1.18.32 61620, connects.

Currently unable to test telnet from opscenter since it is missing yum and yum will not install since it is not compatible with python 2.7.3. Still trying to figure out how to fix that.

Posted 11 months ago #
swoo
Member

ok, got pass the yum install issue.
telnet 10.1.18.32 61621, give me "Connection refused"

Posted 11 months ago #
swoo
Member

netstat -an | grep 61620 on the agent machines show nothing.

Posted 11 months ago #
nickmbailey
Key Master

So the agent won't actually start listening for connections until it is able to establish a connection with the main OpsCenter process itself. We need to debug why the agent is unable to connect to your OpsCenter machine. Are there any other exceptions in the agent log.

I would also recommend switching the agent log to DEBUG, and restarting the process to see if any more information is logged. The logging level is configured in conf/log4j.properties.

Posted 11 months ago #
swoo
Member

Ok, I've set the new log level.
But my system seems to have gone form bad to worse. I had gone to the edit cluster screen, but it won't let me add more hosts and it also won't let me exit the screen. It Error on the screen says "Error creating cluster. Unable to connect to cluster". When I close the error window, it returns me to to the "Welcome to DataStax OpsCenter!" window with now way out. I then killed OpsCenter job, but when I restart and try to get back in through the web, I end up in the same "Welcome to DataStax OpsCenter!" window. I tried to clear the cache, but that didn't do anything. What do I need to clear?

Posted 11 months ago #
thobbs
Administrator

It sounds like you may have wiped out your opscenter config for that cluster when you were editing it. On the "Welcome to DataStax OpsCenter" page, just re-add the info for your cluster. Any metric data you have will still be stored in the cluster, and you'll be able to see it after adding the cluster.

Posted 11 months ago #
swoo
Member

When I try to add a cluster, the log has:

2012-07-12 14:03:37+0000 [] WARN: Unable to find a matching cluster for [u'fe80:0:0:0:225:90ff:fe76:e280%2', u'10.1.18.31', u'0:0:0:0:0:0:0:1%1', u'127.0.0.1']
2012-07-12 14:03:44+0000 [] DEBUG: Trying to connect to node 10.1.18.31
2012-07-12 14:03:44+0000 [] INFO: Starting factory <opscenterd.CassandraService.NoReconnectCassandraClientFactory instance at 0x2176290>
2012-07-12 14:03:44+0000 [] INFO: <twisted.internet.tcp.Connector instance at 0x21767a0> will retry in 3 seconds
2012-07-12 14:03:44+0000 [] DEBUG: Problem while pinging node: User aborted connection: Shutdown requested.
2012-07-12 14:03:44+0000 [] WARN: ProcessingError while calling CreateClusterConfController: Unable to connect to cluster
2012-07-12 14:03:44+0000 [] INFO: Stopping factory <opscenterd.CassandraService.NoReconnectCassandraClientFactory instance at 0x2176290>
2012-07-12 14:03:47+0000 [] DEBUG: Average opscenterd CPU usage: 0.32%, memory usage: 35 MB
2

Posted 11 months ago #
swoo
Member

Killing opscenter and restarting doesn't exit me from the screen.

Posted 11 months ago #
thobbs
Administrator

It appears that opscenter can't connect to your cluster at all via Thrift. Can you try connecting to the cluster with a Thrift client (like cassandra-cli) from the server that you're running opscenter on to verify that there are no connectivity issues?

Posted 11 months ago #
swoo
Member

Should cassandra-cli come with my install? I can't seem to find it.

Posted 11 months ago #

RSS feed for this topic

12 Next »

Reply »

You must log in to post.