Support Forums : DataStax

DataStax Support Forums » OpsCenter

Error in OpsCenter

(15 posts) (4 voices)

Started 1 year ago by blueplastic
Latest reply from nickmbailey
This topic is not resolved

Tags:

No tags yet.

blueplastic
Member

This is probably something simple.

Trying to get OpsCenter to work on my 12 node Brisk-b2 cluster in Amazon.

My Security Group has these ports open:
8888
9160
1024 - 65535 (source is the security group, so only traffic originating within the security group can talk via these ports)

(all of the other ports mentioned in the Brisk AMI on Amazon documentation on this website are also open)

OpsCenter was successfully installed on a Cassandra seed node and the cassandra-env.sh file was changed on all nodes with the updated EC2 public DNS name. Cassandra was also restarted on all nodes.

OpsCenter.conf file:
# opscenterd.conf

[jmx]
cassandra_classpath = /usr/share/opscenter/lib/*.jar
port = 8080

[webserver]
port = 8888
interface = 0.0.0.0
staticdir = /usr/share/opscenter/content

[cassandra]
forget_nodes_after = 2629800 ; average month

# a comma-separated list of places to try for a connection to your Cassandra
# cluster:
seed_hosts = 10.192.x.x

# the API (Thrift) port on your Cassandra cluster
api_port = 9160

[collection]
basic_info_period = 3600
node_poll_period = 120

[authentication]
# if this file does not exist, there will be no password protection. Use the
# set_passwd.py tool (included with OpsCenter) to set passwords.
passwd_file = /etc/opscenter/.passwd

- - - - - - -

When I go to the URL for OpsCenter now, I see Loading Opscenter... then an error:
Error loading cluster: No Cassandra connections available

Here is the Opscenter log (note, IP addresses have been cleaned with x.x):

2011-07-01 00:54:55+0000 [Uninitialized] Thrift pool connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8> failed
Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.

2011-07-01 00:54:55+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 180.96665978431702 left on connect blackout
2011-07-01 00:54:55+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 181.0 seconds.)
2011-07-01 00:54:55+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 180.96680474281311 left on connect blackout
2011-07-01 00:54:55+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 181.0 seconds.)
2011-07-01 00:54:55+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 180.96680688858032 left on connect blackout
2011-07-01 00:54:55+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 181.0 seconds.)
2011-07-01 00:54:55+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 180.96680688858032 left on connect blackout
2011-07-01 00:54:55+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 181.0 seconds.)
2011-07-01 00:54:55+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 180.96680569648743 left on connect blackout
2011-07-01 00:54:55+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 181.0 seconds.)
2011-07-01 00:55:32+0000 [HTTPChannel,10,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:55:32 +0000] "GET /hadoop/jobs HTTP/1.1" 404 72 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:56:32+0000 [HTTPChannel,10,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:56:32 +0000] "GET /hadoop/jobs HTTP/1.1" 404 72 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:32+0000 [HTTPChannel,10,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:32 +0000] "GET /hadoop/jobs HTTP/1.1" 404 72 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:39+0000 [HTTPChannel,10,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:39 +0000] "GET /opscenter/index.html HTTP/1.1" 200 2599 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:39+0000 [HTTPChannel,2,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:39 +0000] "GET /opscenter/js/dojotoolkit/dojox/grid/resources/Grid.css HTTP/1.1" 200 7125 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:39+0000 [HTTPChannel,9,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:39 +0000] "GET /opscenter/css/ripcord.css HTTP/1.1" 200 25573 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:39+0000 [HTTPChannel,14,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:39 +0000] "GET /opscenter/images/opsc-logo.png HTTP/1.1" 200 3052 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:40+0000 [HTTPChannel,10,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:39 +0000] "GET /opscenter/js/dojotoolkit/dijit/themes/tundra/tundra.css HTTP/1.1" 200 76329 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:40+0000 [HTTPChannel,8,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:40 +0000] "GET /opscenter/js/dojotoolkit/dojo/dojo.js HTTP/1.1" 200 90325 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:41+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:41 +0000] "GET /opscenter/js/ripcord/ripcord.js HTTP/1.1" 200 982295 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:42+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:41 +0000] "GET /opscenter/js/dojotoolkit/dojox/gfx/svg.js?1309481862161 HTTP/1.1" 200 39650 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:42+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /opscenter/js/dojotoolkit/dojo/nls/ripcorddojo_en-us.js?1309481862161 HTTP/1.1" 200 5540 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:42+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /opscenter/images/sidebar-line.png HTTP/1.1" 200 291 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:42+0000 [HTTPChannel,8,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /opscenter/images/sidebar-bg.png HTTP/1.1" 200 514 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:42+0000 [HTTPChannel,8,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /rc/userrole HTTP/1.1" 200 4 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:42+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /rc/version HTTP/1.1" 200 7 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:42+0000 [HTTPChannel,13,152.179.x.x] Problem while calling ClusterController
Traceback (most recent call last):
Failure: opscenterd.CassandraService.NoCassandraConnection: No Cassandra connections available

2011-07-01 00:57:42+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /cluster HTTP/1.1" 400 49 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:42+0000 [HTTPChannel,8,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /favicon.ico HTTP/1.1" 404 52 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:43+0000 [HTTPChannel,8,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /opscenter/images/closeicon.png HTTP/1.1" 200 1238 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:43+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /opscenter/js/dojotoolkit/dijit/themes/tundra/images/buttonEnabled.png HTTP/1.1" 200 120 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"
2011-07-01 00:57:56+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8>
2011-07-01 00:57:56+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8>
2011-07-01 00:57:56+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8>
2011-07-01 00:57:56+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8>
2011-07-01 00:57:56+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8>
2011-07-01 00:57:56+0000 [Uninitialized] Thrift pool connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8> failed
Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.

2011-07-01 00:57:56+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 174.66835308074951 left on connect blackout
2011-07-01 00:57:56+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 174.7 seconds.)
2011-07-01 00:57:56+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 174.66850137710571 left on connect blackout
2011-07-01 00:57:56+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 174.7 seconds.)
2011-07-01 00:57:56+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 174.66850113868713 left on connect blackout
2011-07-01 00:57:56+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 174.7 seconds.)
2011-07-01 00:57:56+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 174.66850018501282 left on connect blackout
2011-07-01 00:57:56+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 174.7 seconds.)
2011-07-01 00:57:56+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 174.66850018501282 left on connect blackout
2011-07-01 00:57:56+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 174.7 seconds.)
2011-07-01 00:58:03+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:58:03 +0000] "GET /opscenter/js/dojotoolkit/dijit/themes/tundra/images/buttonHover.png HTTP/1.1" 200 144 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"

Posted 1 year ago #
joaquin
Administrator

For Brisk and Cassandra, you need to switch to port 7199 for JMX.

Could you give that a try and let us know if that works?

Also, OpsCenter must be started before the Cassandra cluster and restarted after every conf change.

Posted 1 year ago #
blueplastic
Member

Port change didn't fix the problem. Could be an issue related to how OpsCenter is installed in our cluster.

OpsCenter was installed on the custom AMI we've created. So, when we spawn a cluster, all nodes technically have OpsCenter installed and the service running. I will try uninstalling OpsCenter from all of the nodes, except one... or at least stop the OpsCenter service on the other nodes.

Also, I'm not sure what kind of problems having OpsCenter installed on all nodes would create. Does each node try to create a new OpsCenter column family?

Anyway, will be on vacation for a week, so I'll have to pick back up on this issue on July 13th, after the Cassandra summit.

Posted 1 year ago #
joaquin
Administrator

Okay, well enjoy your vacation and we will definitely pick this up after the summit.

And yes, OpsCenter should only be installed on one machine, preferable a separate machine. With the AMI code, I made it check the indexid was 0 before installing. You could also do something like this to see if it helps lower the overall load.

Posted 1 year ago #
blueplastic
Member

I decided to completely uninstall the older version of OpsCenter and moved to the new version released a couple of days ago. The new version of installed on just one of the seed nodes.

When I load the web GUI for OpsCenter, I get this error:

Error Loading OpsCenter
OpsCenter is having trouble gathering basic information about your cluster. This usually means OpsCenter cannot connect to your cluster via thrift.

Be sure to check your seed hosts in opscenterd.conf.

----

Error: No Cassandra connections available

The opscenterd.conf file correctly has the seeds set as the private IPs for my Amazon EC2 nodes:

# opscenterd.conf

[jmx]
# The default jmx port for Cassandra >= 0.8.0 is 7199. If you are using
# Cassandra 0.7.*, the default is 8080, and you should change this to
# reflect that.
port = 7199

[webserver]
port = 8888
interface = 0.0.0.0

[logging]
# level may be TRACE, DEBUG, INFO, WARN, or ERROR
level = INFO

[cassandra]
# a comma-separated list of places to try for a connection to your Cassandra
# cluster:
seed_hosts = 10.192.x.x,10.114.x.x

# the API (Thrift) port on your Cassandra cluster
api_port = 9160

[authentication]
# if this file does not exist, there will be no password protection. Use the
# set_passwd.py tool (included with OpsCenter) to set passwords.
passwd_file = /etc/opscenter/.passwd

[cassandra_metrics]
# A list of keyspaces to *not* collect metrics for, separated by commas
ignored_keyspaces = system, OpsCenter

# A list of column families to *not* collect metrics for, separated
# by commas. Each entry should be of the form "ks.cf". For example:
# metrics_ignored_column_families = system.NodeInfo, system.Schema, Keyspace1.Standard1
ignored_column_families =

OpsCenter log shows:

2011-07-12 23:05:09+0000 [-] Log opened.
2011-07-12 23:05:09+0000 [-] twistd 10.2.0 (/usr/bin/python 2.6.6) starting up.
2011-07-12 23:05:09+0000 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2011-07-12 23:05:09+0000 [-] set uid/gid 0/0
2011-07-12 23:05:09+0000 [-] Logging level set to 'info'
2011-07-12 23:05:09+0000 [-] OpsCenterdService startService
2011-07-12 23:05:09+0000 [-] OpsCenter version: 1.2
2011-07-12 23:05:09+0000 [-] Compatible agent version: 2.0
2011-07-12 23:05:09+0000 [-] HTTP BASIC authentication disabled
2011-07-12 23:05:09+0000 [-] agent_config items: {'jmx_user': '', 'jmx_pass': '', 'jmx_port': 7199, 'metrics_enabled': 1, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter'}
2011-07-12 23:05:09+0000 [-] opscenterd.WebServer.OpsCenterdWebServer starting on 8888
2011-07-12 23:05:09+0000 [-] Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x308cb00>
2011-07-12 23:05:09+0000 [-] JmxJythonService started. State is now "startup".
2011-07-12 23:05:09+0000 [-] Starting CassandraCluster service
2011-07-12 23:05:09+0000 [-] morbid.morbid.StompFactory starting on 61619
2011-07-12 23:05:09+0000 [-] Starting factory <morbid.morbid.StompFactory instance at 0x3098878>
2011-07-12 23:05:09+0000 [-] morbid.morbid.StompFactory starting on 61620
2011-07-12 23:05:09+0000 [-] Starting PushService
2011-07-12 23:05:09+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)
2011-07-12 23:05:09+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)
2011-07-12 23:05:09+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.)

[there's tons of lines like the one above, which I removed]

2011-07-12 23:05:16+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 6.8 seconds.)
2011-07-12 23:05:16+0000 [-] JmxJythonService got jmxjython startup message. State is now "normal".
2011-07-12 23:05:22+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 3.6 seconds.)

2011-07-12 23:09:39+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 160.6 seconds.)
2011-07-12 23:09:39+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 160.6 seconds.)
2011-07-12 23:10:37+0000 [HTTPChannel,3,152.179.x.x] Problem while calling ClusterController
Traceback (most recent call last):
Failure: opscenterd.CassandraService.NoCassandraConnection: No Cassandra connections available

All the ports I mentioned in the first post are still open in the EC2 security group.

Would it help if I turned on higher level debugging in OpsCenter?

Any ideas?

Posted 1 year ago #
blueplastic
Member

Quick update.

I also tried temporarily opening all TCP ports in my EC2 Security Group (source: 0.0.0.0/0).

Then I restarted OpsCenter:

root@domU-12-31-39-0E-x-x:/home/ubuntu# sudo service opscenterd restart
* Restarting Cassandra cluster manager opscenterd [ OK ]

Finally, I refreshed the OpsCenter website but got the same error.

I have removed the all TCP rule from my security group for now.

Also, I want to clarify that all of my 12 Cassandra nodes are technically in the same Amazon region and two different Availability Zones. However, we’re using the Property Files snitch and simulating two different DCs and the 1st DC has two Racs.

Here are all of the TCP ports that are current open in my security group:

1024 - 65535 sg-b04c3dd9 (BriskNodes)
22 (SSH) 0.0.0.0/0
80 (HTTP) 0.0.0.0/0
7000 0.0.0.0/0
7199 0.0.0.0/0
8012 0.0.0.0/0
8888 0.0.0.0/0
8983 0.0.0.0/0
9160 0.0.0.0/0
50030 0.0.0.0/0
50060 0.0.0.0/0

Posted 1 year ago #
mbulman
Key Master

Looks like OpsCenter can't talk to your nodes via thrift. Can you trying telnet <IP> 9160 from the opscenter machine to the IPs listed in seed_hosts in opscenterd.conf?

Posted 1 year ago #
nickmbailey
Key Master

The problem is definitely with thrift.

From the logs you posted it doesn't look like OpsCenter is even trying to connect to the hosts listed in the conf. The conf you posted looks normal at first glance, but perhaps there is a syntax error in there somewhere?

I would recommend turning on DEBUG logging and seeing if that shows any more useful errors regarding thrift.

Posted 1 year ago #
blueplastic
Member

Thanks for the advice, Mike and Nick.

So, the first node in my 12 node cluster is a seed node and the OpsCenter server. The 8th node, which is in a different logical DC is the other seed node.

I logged into the 1st node/seed (the OpsCenter server) and was able to telnet into both seeds:

ubuntu@domU-12-31-39-0E-x-x:~$ telnet 10.192.x.x 9160
Trying 10.192.x.x...
Connected to 10.192.x.x.
Escape character is '^]'.

ubuntu@domU-12-31-39-0E-x-x:~$ telnet 10.114.x.x 9160
Trying 10.114.x.x...
Connected to 10.114.x.x.
Escape character is '^]'.

Here is the output from DEBUG turned on:

2011-07-13 21:04:29+0000 [-] Log opened.
2011-07-13 21:04:29+0000 [-] twistd 10.2.0 (/usr/bin/python 2.6.6) starting up.
2011-07-13 21:04:29+0000 [-] reactor class: twisted.internet.selectreactor.SelectReactor.
2011-07-13 21:04:29+0000 [-] set uid/gid 0/0
2011-07-13 21:04:29+0000 [-] Logging level set to 'debug'
2011-07-13 21:04:29+0000 [-] OpsCenterdService startService
2011-07-13 21:04:29+0000 [-] OpsCenter version: 1.2
2011-07-13 21:04:29+0000 [-] Compatible agent version: 2.0
2011-07-13 21:04:29+0000 [-] HTTP BASIC authentication disabled
2011-07-13 21:04:29+0000 [-] agent_config items: {'jmx_user': '', 'jmx_pass': '', 'jmx_port': 7199, 'metrics_enabled': 1, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter'}
2011-07-13 21:04:29+0000 [-] opscenterd.WebServer.OpsCenterdWebServer starting on 8888
2011-07-13 21:04:29+0000 [-] Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x1876b00>
2011-07-13 21:04:29+0000 [-] Jython args: ['/usr/bin/java', '-Djava.library.path=/usr/lib/jni', '-Dpython.cachedir=/root/.jython_cache', 'org.python.util.jython', '-jar', '/usr/share/opscenter/bin/jmxjython.jar']
2011-07-13 21:04:29+0000 [-] JmxJythonService started. State is now "startup".
2011-07-13 21:04:29+0000 [-] Starting CassandraCluster service
2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638>
2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8>
2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638>
2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8>
2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638>
2011-07-13 21:04:29+0000 [-] Waiting for describe_version() results
2011-07-13 21:04:29+0000 [-] morbid.morbid.StompFactory starting on 61619
2011-07-13 21:04:29+0000 [-] Starting factory <morbid.morbid.StompFactory instance at 0x1882878>
2011-07-13 21:04:29+0000 [-] morbid.morbid.StompFactory starting on 61620
2011-07-13 21:04:29+0000 [-] Starting PushService
2011-07-13 21:04:29+0000 [-] Waiting for a successful thrift call
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Thrift pool connection to <CassandraNode 10.192.x.x:9160 @0x1886638> failed
Traceback (most recent call last):
Failure: thrift.Thrift.TApplicationException: Internal error processing describe_ring

2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.10830187797546387 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8>
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.10861492156982422 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8>
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.10861301422119141 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8>
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Thrift pool connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> failed
Traceback (most recent call last):
Failure: thrift.Thrift.TApplicationException: Internal error processing describe_ring

2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.11836552619934082 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.10730290412902832 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.)
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.11850237846374512 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.10667991638183594 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.)
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.1185004711151123 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.097223997116088867 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.)
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.11850333213806152 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.096638917922973633 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.)
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.11850428581237793 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.096105098724365234 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.)
2011-07-13 21:04:29+0000 [-] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.015861272811889648 left on connect blackout
2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638>
2011-07-13 21:04:29+0000 [-] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.015218257904052734 left on connect blackout
2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638>
2011-07-13 21:04:29+0000 [-] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.014885425567626953 left on connect blackout
2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638>
2011-07-13 21:04:29+0000 [-] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.014558315277099609 left on connect blackout
2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638>
2011-07-13 21:04:29+0000 [-] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.014231443405151367 left on connect blackout
2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638>
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Thrift pool connection to <CassandraNode 10.192.x.x:9160 @0x1886638> failed
Traceback (most recent call last):
Failure: thrift.Thrift.TApplicationException: Internal error processing describe_ring

2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.26951456069946289 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8>
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.26970458030700684 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8>
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.26970577239990234 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8>
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.26970577239990234 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8>
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.26970577239990234 left on connect blackout
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8>
2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Thrift pool connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> failed
Traceback (most recent call last):
Failure: thrift.Thrift.TApplicationException: Internal error processing describe_ring

Posted 1 year ago #
nickmbailey
Key Master

Your problem is:

Failure: thrift.Thrift.TApplicationException: Internal error processing describe_ring

Unfortunately thats all the information we get back from cassandra, however if you check the cassandra logs you should see an indication of what the problem is when processing describe_ring. Often times it is a NetworkTopologyStrategy/datacenter misconfiguration.

Posted 1 year ago #
mbulman
Key Master

Failure: thrift.Thrift.TApplicationException: Internal error processing describe_ring

This line indicates there is some sort of problem with the nodes you're trying to connect to, most likely a schema disagreement. If you look in the cassandra log on that server, you should be able to get more info.

Posted 1 year ago #

blueplastic
Member

Hmm, you were right Nick.

So, I started doing some investigation, but can't find the root cause of the problem here.

The Cassandra system.log is littered with these errors, like you predicted:

ERROR [pool-2-thread-10274] 2011-07-12 23:43:40,320 Cassandra.java (line 3668) Internal error processing describe_ring
java.lang.IllegalStateException: datacenter (Brisk) has no more endpoints, (1) replicas still needed
	at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:124)
	at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:100)
	at org.apache.cassandra.service.StorageService.constructRangeToEndpointMap(StorageService.java:618)
	at org.apache.cassandra.service.StorageService.getRangeToAddressMap(StorageService.java:593)
	at org.apache.cassandra.service.StorageService.getRangeToEndpointMap(StorageService.java:567)
	at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:714)
	at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664)
	at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464)
	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

So, it sounds like the brisk DC needs another replica? I don't fully understand this.

I remember we got rid of this error a couple of weeks ago by clearing out /var/lib/cassandra, but we can't do that now b/c there is data on this cluster.

Here is what nodetool ring shows (note, I reorganized the row output according to DC/RAC):

Address         DC          Rack        Status State   Load            Owns    Token
                                                                               148873535527910577765226390751398592512
10.192.143.x   DC1         RAC1        Up     Normal  643.42 GB       12.50%  0
10.192.171.x  DC1         RAC1        Up     Normal  128.96 GB       6.25%   21267647932558653966460912964485513216
10.210.95.x   DC1         RAC1        Up     Normal  128.34 GB       12.50%  42535295865117307932921825928971026432
10.211.19.x    DC1         RAC1        Up     Normal  128.55 GB       6.25%   63802943797675961899382738893456539648
10.68.58.x     DC1         RAC2        Up     Normal  643.05 GB       12.50%  85070591730234615865843651857942052864
10.110.31.x    DC1         RAC2        Up     Normal  128.84 GB       6.25%   106338239662793269832304564822427566080
10.96.58.x    DC1         RAC2        Up     Normal  128.11 GB       12.50%  127605887595351923798765477786913079296
10.210.195.x   DC1         RAC2        Up     Normal  129.33 GB       6.25%   148873535527910577765226390751398592512
10.114.138.x  DC2         RAC1        Up     Normal  258.04 GB       6.25%   10633823966279326983230456482242756608
10.203.79.x   DC2         RAC1        Up     Normal  257.14 GB       6.25%   53169119831396634916152282411213783040
10.242.209.x  DC2         RAC1        Up     Normal  256.58 GB       6.25%   95704415696513942849074108340184809472
10.38.25.x    DC2         RAC1        Up     Normal  257.08 GB       6.25%   138239711561631250781995934269155835904

YAML file snitch settings:
endpoint_snitch: org.apache.cassandra.locator.PropertyFileSnitch
dynamic_snitch: true

cassandra-topology.properties file:

# Cassandra Node IP=Data Center:Rack
10.192.143.x=DC1:RAC1
10.192.171.x=DC1:RAC1
10.210.95.x=DC1:RAC1
10.211.19.x=DC1:RAC1
10.68.58.x=DC1:RAC2
10.110.31.x=DC1:RAC2
10.96.58.x=DC1:RAC2
10.210.195.x=DC1:RAC2
10.114.138.x=DC2:RAC1
10.203.79.x=DC2:RAC1
10.242.209.x=DC2:RAC1
10.38.25.x=DC2:RAC1

# default for unknown nodes
default=DC1:RAC1
# Native IPv6 is supported, however you must escape the colon in the IPv6 Address
# Also be sure to comment out JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"
# in cassandra-env.sh
#fe80\:0\:0\:0\:202\:b3ff\:fe1e\:8329=DC1:RAC3

Our Keyspace is set to NetworkTopologyStrategy:

[default@DemoKS] describe keyspace;
Keyspace: DemoKS:
  Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
  Durable Writes: true
    Options: [DC2:1, DC1:2]
  Column Families:
    ColumnFamily: DemoCF
      Key Validation Class: org.apache.cassandra.db.marshal.BytesType
      Default column value validator: org.apache.cassandra.db.marshal.BytesType
      Columns sorted by: org.apache.cassandra.db.marshal.BytesType
      Row cache size / save period in seconds: 0.0/0
      Key cache size / save period in seconds: 200000.0/14400
      Memtable thresholds: 1.0875/232/1440 (millions of ops/MB/minutes)
      GC grace seconds: 864000
      Compaction min/max thresholds: 4/32
      Read repair chance: 1.0
      Replicate on write: false
      Built indexes: []

Posted 1 year ago #

nickmbailey
Key Master

So by default the brisk specific column families are created with NetworkTopologyStrategy and set to only exist in the 'Brisk' datacenter. When you use the BriskSnitch that datacenter gets set up automatically. Since you are using the property file snitch, and have no nodes in the 'Brisk' datacenter, you are seeing those errors.

You can either modify the brisk column families to replicate to 'DC1' or 'DC2', or just change the name of one of those datacenters.

Make sense?

Posted 1 year ago #
blueplastic
Member

Yup, you're right Nick. Thanks for the help, don't think I would have figured this out on my own.

The brisk_system keyspace was automatically set up to place one replica in the Brisk data center (which doesn't exist since I'm not using BriskSnitch):
[default@brisk_system] describe keyspace;
Keyspace: brisk_system:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Durable Writes: true
Options: [Brisk:1, Cassandra:0]

I ran this command to update the keyspace:
[default@brisk_system] update keyspace brisk_system with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options=[{DC2:1, DC1:0}];
5a8ec1c0-ae40-11e0-0000-8b19d956fdfe
Waiting for schema agreement...
... schemas agree across the cluster

And finally:

[default@brisk_system] describe keyspace;
Keyspace: brisk_system:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Durable Writes: true
Options: [DC2:1, DC1:0]

OpsCenter works now and I just auto-installed the agent on the rest of the nodes.

By the way, OpsCenter v1.2 is pretty sweet.

Posted 1 year ago #
nickmbailey
Key Master

Good to hear it all got figured out. Glad you like the newest version :).

Posted 1 year ago #

RSS feed for this topic

Reply

You must log in to post.