DataStax Support Forums » User Favorites: hhj

DataStax Support Forums » User Favorites: hhj hhj Software, Support, and Training for Apache Cassandra en-US Sat, 25 May 2013 16:57:58 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php hhj on "Compatible with Ubuntu 12.10?" http://www.datastax.com/support-forums/topic/compatible-with-ubuntu-1210#post-7580 Mon, 19 Nov 2012 22:41:59 +0000 hhj 7580@http://www.datastax.com/support-forums/ Now that made a huge difference. All the agent nodes are now responding to requests. Thanks a lot for your help. nickmbailey on "Compatible with Ubuntu 12.10?" http://www.datastax.com/support-forums/topic/compatible-with-ubuntu-1210#post-7579 Mon, 19 Nov 2012 21:53:59 +0000 nickmbailey 7579@http://www.datastax.com/support-forums/ The agents won't listen on that port until the receive a response from OpsCenter. The problem appears to be that OpsCenter can't match the agents to a cluster. You can see the relevant log line: 2012-11-19 20:48:17+0000 [] WARN: Unable to find a matching cluster for [u'0:0:0:0:0:0:0:1%1', u'fe80:0:0:0:1031:3cff:fe06:5878%2', u'127.0.0.1', u'10.51.91.134'] It appears that your cassandra nodes are configured with an interface that the agent can't auto detect. You can resolve this by adding an additional configuration line to address.yaml that specifies the broadcast/listen address that the cassandra node is using: local_interface: <cassandra listen address> hhj on "Compatible with Ubuntu 12.10?" http://www.datastax.com/support-forums/topic/compatible-with-ubuntu-1210#post-7578 Mon, 19 Nov 2012 21:24:41 +0000 hhj 7578@http://www.datastax.com/support-forums/ As mentioned in the initial post, the agents aren't even listening on port 61621 on the nodes, which I guess is the root cause of the problems. hhj on "Compatible with Ubuntu 12.10?" http://www.datastax.com/support-forums/topic/compatible-with-ubuntu-1210#post-7577 Mon, 19 Nov 2012 21:04:48 +0000 hhj 7577@http://www.datastax.com/support-forums/ Cassandra version is 1.1.6. Opscenter 2.1.2. I have not changed any defaults other than use_ssl: no. The setup is running in AWS across multiple regions. <code> tail /var/log/opscenter/opscenterd.log 2012-11-19 20:48:16+0000 [PING] WARN: Unable to collect datacenter, rack information: Failed query to <a href="http://23.21.197.95:61621/cluster/datacenter?node_ip=176.34.238.192" rel="nofollow">http://23.21.197.95:61621/cluster/datacenter?node_ip=176.34.238.192</a> : Connection was refused by other side: 111: Connection refused. 2012-11-19 20:48:16+0000 [PING] WARN: HTTP request <a href="http://54.251.40.92:61621/cluster/datacenter?node_ip=23.21.197.95" rel="nofollow">http://54.251.40.92:61621/cluster/datacenter?node_ip=23.21.197.95</a> failed: Connection was refused by other side: 111: Connection refused. 2012-11-19 20:48:16+0000 [PING] WARN: Unable to collect datacenter, rack information: Failed query to <a href="http://54.251.40.92:61621/cluster/datacenter?node_ip=23.21.197.95" rel="nofollow">http://54.251.40.92:61621/cluster/datacenter?node_ip=23.21.197.95</a> : Connection was refused by other side: 111: Connection refused. 2012-11-19 20:48:16+0000 [PING] WARN: HTTP request <a href="http://54.251.40.92:61621/cluster/datacenter?node_ip=54.251.40.92" rel="nofollow">http://54.251.40.92:61621/cluster/datacenter?node_ip=54.251.40.92</a> failed: Connection was refused by other side: 111: Connection refused. 2012-11-19 20:48:16+0000 [PING] WARN: Unable to collect datacenter, rack information: Failed query to <a href="http://54.251.40.92:61621/cluster/datacenter?node_ip=54.251.40.92" rel="nofollow">http://54.251.40.92:61621/cluster/datacenter?node_ip=54.251.40.92</a> : Connection was refused by other side: 111: Connection refused. 2012-11-19 20:48:17+0000 [] WARN: Unable to find a matching cluster for [u'0:0:0:0:0:0:0:1%1', u'fe80:0:0:0:1031:3cff:fe06:5878%2', u'127.0.0.1', u'10.51.91.134'] 2012-11-19 20:48:27+0000 [] WARN: Unable to find a matching cluster for [u'10.142.214.245', u'0:0:0:0:0:0:0:1%1', u'fe80:0:0:0:1031:41ff:fe04:d507%2', u'127.0.0.1'] 2012-11-19 20:48:37+0000 [] WARN: Unable to find a matching cluster for [u'0:0:0:0:0:0:0:1%1', u'fe80:0:0:0:1031:3cff:fe06:5878%2', u'127.0.0.1', u'10.51.91.134'] </code> nickmbailey on "Compatible with Ubuntu 12.10?" http://www.datastax.com/support-forums/topic/compatible-with-ubuntu-1210#post-7575 Mon, 19 Nov 2012 16:22:58 +0000 nickmbailey 7575@http://www.datastax.com/support-forums/ A couple of questions * What version of cassandra are you using * Did you change the default jmx port * Can you attach the main opscenter log? It looks like the agents are connecting to it but it is not responding. hhj on "Compatible with Ubuntu 12.10?" http://www.datastax.com/support-forums/topic/compatible-with-ubuntu-1210#post-7572 Mon, 19 Nov 2012 13:04:50 +0000 hhj 7572@http://www.datastax.com/support-forums/ Thanks for the reply. Yes, SSL is disabled through use_ssl: 0. There are no messages regarding the agents in the opscenterd logs, other than the WARN described above. thobbs on "Compatible with Ubuntu 12.10?" http://www.datastax.com/support-forums/topic/compatible-with-ubuntu-1210#post-7540 Fri, 16 Nov 2012 23:02:15 +0000 thobbs 7540@http://www.datastax.com/support-forums/ How did you disable SSL on the agent? The correct method is to add the following to <code>address.yaml</code>: "<code>use_ssl: 0</code>". (The <code>address.yaml</code> file is in <code>/var/lib/opscenter-agent/conf</code> if it was installed through the OpsCenter UI or from a package.) Do the opscenterd logs show any messages coming in from that agent? hhj on "Compatible with Ubuntu 12.10?" http://www.datastax.com/support-forums/topic/compatible-with-ubuntu-1210#post-7507 Thu, 15 Nov 2012 19:10:09 +0000 hhj 7507@http://www.datastax.com/support-forums/ Does OpsCenter have any known issues on Ubuntu 12.10? The reason I'm asking, is that on every node I'm seeing the following messages in the agent logs: <code> DEBUG [pool-2-thread-2] 2012-11-15 19:01:05,819 Sending message '[null,["2.10","\/453709357\/conf",["0:0:0:0:0:0:0:1%1","fe80:0:0:0:1031:3cff:fe06:5878%2","127.0.0.1","10.51.91.134"]]]' to channel '/conf-request'. INFO [Initialization] 2012-11-15 19:01:05,820 Sleeping for 20s before trying to determine IP over JMX again DEBUG [pool-2-thread-2] 2012-11-15 19:01:25,822 Sending message '[null,["2.10","\/453709357\/conf",["0:0:0:0:0:0:0:1%1","fe80:0:0:0:1031:3cff:fe06:5878%2","127.0.0.1","10.51.91.134"]]]' to channel '/conf-request'. INFO [Initialization] 2012-11-15 19:01:25,823 Sleeping for 20s before trying to determine IP over JMX again DEBUG [pool-2-thread-2] 2012-11-15 19:01:45,825 Sending message '[null,["2.10","\/453709357\/conf",["0:0:0:0:0:0:0:1%1","fe80:0:0:0:1031:3cff:fe06:5878%2","127.0.0.1","10.51.91.134"]]]' to channel '/conf-request'. INFO [Initialization] 2012-11-15 19:01:45,827 Sleeping for 20s before trying to determine IP over JMX again DEBUG [pool-2-thread-2] 2012-11-15 19:02:05,828 Sending message '[null,["2.10","\/453709357\/conf",["0:0:0:0:0:0:0:1%1","fe80:0:0:0:1031:3cff:fe06:5878%2","127.0.0.1","10.51.91.134"]]]' to channel '/conf-request'. INFO [Initialization] 2012-11-15 19:02:05,830 Sleeping for 20s before trying to determine IP over JMX again DEBUG [pool-2-thread-2] 2012-11-15 19:02:25,832 Sending message '[null,["2.10","\/453709357\/conf",["0:0:0:0:0:0:0:1%1","fe80:0:0:0:1031:3cff:fe06:5878%2","127.0.0.1","10.51.91.134"]]]' to channel '/conf-request'. INFO [Initialization] 2012-11-15 19:02:25,833 Sleeping for 20s before trying to determine IP over JMX again </code> The agents doesn't seem to be listening on port 61621, as that port doesn't appear in netstat. This results in the following messages on the opscenter server: <code>WARN: Unable to collect datacenter, rack information: Failed query to <a href="http://x.x.x.x:61621/cluster/datacenter?node_ip=x.x.x.x" rel="nofollow">http://x.x.x.x:61621/cluster/datacenter?node_ip=x.x.x.x</a> : The http request to the agent timed out. This most likely indicates that you need to restart the agent on this node.</code> Connectivity between the servers on port 61620/61621 has been verified. Has been tested on Oracle Java JRE 1.6 and OpenJDK JRE 1.7. SSL has been disabled, as part of my debugging efforts.