Support Forums : DataStax

DataStax Support Forums » OpsCenter

Compatible with Ubuntu 12.10?

(8 posts) (3 voices)

Started 6 months ago by hhj
Latest reply from hhj
This topic is resolved

Tags:

No tags yet.

hhj
Member

Does OpsCenter have any known issues on Ubuntu 12.10?

The reason I'm asking, is that on every node I'm seeing the following messages in the agent logs:
DEBUG [pool-2-thread-2] 2012-11-15 19:01:05,819 Sending message '[null,["2.10","\/453709357\/conf",["0:0:0:0:0:0:0:1%1","fe80:0:0:0:1031:3cff:fe06:5878%2","127.0.0.1","10.51.91.134"]]]' to channel '/conf-request'. INFO [Initialization] 2012-11-15 19:01:05,820 Sleeping for 20s before trying to determine IP over JMX again DEBUG [pool-2-thread-2] 2012-11-15 19:01:25,822 Sending message '[null,["2.10","\/453709357\/conf",["0:0:0:0:0:0:0:1%1","fe80:0:0:0:1031:3cff:fe06:5878%2","127.0.0.1","10.51.91.134"]]]' to channel '/conf-request'. INFO [Initialization] 2012-11-15 19:01:25,823 Sleeping for 20s before trying to determine IP over JMX again DEBUG [pool-2-thread-2] 2012-11-15 19:01:45,825 Sending message '[null,["2.10","\/453709357\/conf",["0:0:0:0:0:0:0:1%1","fe80:0:0:0:1031:3cff:fe06:5878%2","127.0.0.1","10.51.91.134"]]]' to channel '/conf-request'. INFO [Initialization] 2012-11-15 19:01:45,827 Sleeping for 20s before trying to determine IP over JMX again DEBUG [pool-2-thread-2] 2012-11-15 19:02:05,828 Sending message '[null,["2.10","\/453709357\/conf",["0:0:0:0:0:0:0:1%1","fe80:0:0:0:1031:3cff:fe06:5878%2","127.0.0.1","10.51.91.134"]]]' to channel '/conf-request'. INFO [Initialization] 2012-11-15 19:02:05,830 Sleeping for 20s before trying to determine IP over JMX again DEBUG [pool-2-thread-2] 2012-11-15 19:02:25,832 Sending message '[null,["2.10","\/453709357\/conf",["0:0:0:0:0:0:0:1%1","fe80:0:0:0:1031:3cff:fe06:5878%2","127.0.0.1","10.51.91.134"]]]' to channel '/conf-request'. INFO [Initialization] 2012-11-15 19:02:25,833 Sleeping for 20s before trying to determine IP over JMX again
The agents doesn't seem to be listening on port 61621, as that port doesn't appear in netstat.

This results in the following messages on the opscenter server:
WARN: Unable to collect datacenter, rack information: Failed query to http://x.x.x.x:61621/cluster/datacenter?node_ip=x.x.x.x : The http request to the agent timed out. This most likely indicates that you need to restart the agent on this node.

Connectivity between the servers on port 61620/61621 has been verified.
Has been tested on Oracle Java JRE 1.6 and OpenJDK JRE 1.7.
SSL has been disabled, as part of my debugging efforts.

Posted 6 months ago #
thobbs
Administrator

How did you disable SSL on the agent? The correct method is to add the following to address.yaml: "use_ssl: 0". (The address.yaml file is in /var/lib/opscenter-agent/conf if it was installed through the OpsCenter UI or from a package.)

Do the opscenterd logs show any messages coming in from that agent?

Posted 6 months ago #
hhj
Member

Thanks for the reply.

Yes, SSL is disabled through use_ssl: 0.

There are no messages regarding the agents in the opscenterd logs, other than the WARN described above.

Posted 6 months ago #
nickmbailey
Key Master

A couple of questions

* What version of cassandra are you using
* Did you change the default jmx port
* Can you attach the main opscenter log? It looks like the agents are connecting to it but it is not responding.

Posted 6 months ago #
hhj
Member

Cassandra version is 1.1.6. Opscenter 2.1.2.
I have not changed any defaults other than use_ssl: no.
The setup is running in AWS across multiple regions.

tail /var/log/opscenter/opscenterd.log 2012-11-19 20:48:16+0000 [PING] WARN: Unable to collect datacenter, rack information: Failed query to http://23.21.197.95:61621/cluster/datacenter?node_ip=176.34.238.192 : Connection was refused by other side: 111: Connection refused. 2012-11-19 20:48:16+0000 [PING] WARN: HTTP request http://54.251.40.92:61621/cluster/datacenter?node_ip=23.21.197.95 failed: Connection was refused by other side: 111: Connection refused. 2012-11-19 20:48:16+0000 [PING] WARN: Unable to collect datacenter, rack information: Failed query to http://54.251.40.92:61621/cluster/datacenter?node_ip=23.21.197.95 : Connection was refused by other side: 111: Connection refused. 2012-11-19 20:48:16+0000 [PING] WARN: HTTP request http://54.251.40.92:61621/cluster/datacenter?node_ip=54.251.40.92 failed: Connection was refused by other side: 111: Connection refused. 2012-11-19 20:48:16+0000 [PING] WARN: Unable to collect datacenter, rack information: Failed query to http://54.251.40.92:61621/cluster/datacenter?node_ip=54.251.40.92 : Connection was refused by other side: 111: Connection refused. 2012-11-19 20:48:17+0000 [] WARN: Unable to find a matching cluster for [u'0:0:0:0:0:0:0:1%1', u'fe80:0:0:0:1031:3cff:fe06:5878%2', u'127.0.0.1', u'10.51.91.134'] 2012-11-19 20:48:27+0000 [] WARN: Unable to find a matching cluster for [u'10.142.214.245', u'0:0:0:0:0:0:0:1%1', u'fe80:0:0:0:1031:41ff:fe04:d507%2', u'127.0.0.1'] 2012-11-19 20:48:37+0000 [] WARN: Unable to find a matching cluster for [u'0:0:0:0:0:0:0:1%1', u'fe80:0:0:0:1031:3cff:fe06:5878%2', u'127.0.0.1', u'10.51.91.134']

Posted 6 months ago #
hhj
Member

As mentioned in the initial post, the agents aren't even listening on port 61621 on the nodes, which I guess is the root cause of the problems.

Posted 6 months ago #
nickmbailey
Key Master

The agents won't listen on that port until the receive a response from OpsCenter. The problem appears to be that OpsCenter can't match the agents to a cluster. You can see the relevant log line:

2012-11-19 20:48:17+0000 [] WARN: Unable to find a matching cluster for [u'0:0:0:0:0:0:0:1%1', u'fe80:0:0:0:1031:3cff:fe06:5878%2', u'127.0.0.1', u'10.51.91.134']

It appears that your cassandra nodes are configured with an interface that the agent can't auto detect. You can resolve this by adding an additional configuration line to address.yaml that specifies the broadcast/listen address that the cassandra node is using:

local_interface: <cassandra listen address>

Posted 6 months ago #
hhj
Member

Now that made a huge difference.
All the agent nodes are now responding to requests.

Thanks a lot for your help.

Posted 6 months ago #

RSS feed for this topic

Reply

You must log in to post.