resolved
Anonymous
Unregistered
Hi and thanks for this nice tool,
I have just upgraded a 2 node cassandra cluster to version 1.0.6. The 1.0.5 version worked fine with the OpsCenter, but now the OpsCenter always returns me the message: "OpsCenter has not established a connection to your Cassandra cluster." Are they not compatible with each other?
Thanks & best regards
Daniel
Daniel,
OpsCenter should work with 1.0.6. It would help if you could post the OpsCenter log so we can diagnose what is causing thrift communication problems. It should be located at /var/log/opscenter/opscenterd.log.
Anonymous
Unregistered
It's always showing:
2012-01-05 19:05:43+0100 [] INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 182.0 seconds.)
when I 'ring' the nodes via nodetool all seems fine:
Address DC Rack Status State Load Owns Token
85070591730234615865843651857942052864
xxx.xxx.xxx.xxx datacenter1 rack1 Up Normal 277.53 MB 50.00% 0
xxx.xxx.xxx.xxx datacenter1 rack1 Up Normal 285.68 MB 50.00% 85070591730234615865843651857942052864
Daniel,
That error from the log is the connection pool attempting to fill the pool. I believe there should be errors before that in the log indicating the initial connection failure. You may want to restart the OpsCenter process and then check for errors as the process starts up.
Anonymous
Unregistered
OK, I have restarted several times. And I have installed a fresh cassandra cluster with no data etc and the log also doesn't show any error I think:
2012-01-05 20:45:41+0100 [] INFO: Server Shut Down.
2012-01-05 20:45:42+0100 [] INFO: Log opened.
2012-01-05 20:45:42+0100 [] INFO: twistd 10.2.0 (/usr/bin/python2.6 2.6.6) starting up.
2012-01-05 20:45:42+0100 [] INFO: reactor class: twisted.internet.selectreactor.SelectReactor.
2012-01-05 20:45:42+0100 [] INFO: set uid/gid 0/0
2012-01-05 20:45:42+0100 [] INFO: Logging level set to 'info'
2012-01-05 20:45:42+0100 [] INFO: OpsCenterdService startService
2012-01-05 20:45:42+0100 [] INFO: OpsCenter version: 1.3.1
2012-01-05 20:45:42+0100 [] INFO: Compatible agent version: 2.4
2012-01-05 20:45:42+0100 [] INFO: HTTP BASIC authentication disabled
2012-01-05 20:45:42+0100 [] INFO: agent_config items: {'metrics_enabled': 1, 'http_server_port': 61621, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter', 'jmx_port': 7199}
2012-01-05 20:45:42+0100 [] INFO: opscenterd.WebServer.OpsCenterdWebServer starting on 8888
2012-01-05 20:45:42+0100 [] INFO: Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x2a28a70>
2012-01-05 20:45:42+0100 [] INFO: Starting CassandraCluster service
2012-01-05 20:45:42+0100 [] INFO: morbid.morbid.StompFactory starting on 61619
2012-01-05 20:45:42+0100 [] INFO: Starting factory <morbid.morbid.StompFactory instance at 0x2a4f200>
2012-01-05 20:45:42+0100 [] INFO: morbid.morbid.StompFactory starting on 61620
2012-01-05 20:45:42+0100 [] INFO: Starting PushService
2012-01-05 20:45:42+0100 [] INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.)
2012-01-05 20:45:42+0100 [] INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.)
2012-01-05 20:45:42+0100 [] INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.)
2012-01-05 20:45:42+0100 [] INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.)
2012-01-05 20:45:42+0100 [] INFO: (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.)
2012-01-05 20:45:42+0100 [] INFO: CPU Info: ['2800.000', '2800.000']
2012-01-05 20:45:42+0100 [] INFO: Mem Info: 3934MB
2012-01-05 20:45:42+0100 [] INFO: OS Version: Linux version 2.6.32-5-amd64 (Debian 2.6.32-34squeeze1) (dannf@debian.org) (gcc version 4.3.5 (Debian 4.3.5-4) ) #1 SMP Wed May 18 23:13:22 UTC 2011
2012-01-05 20:45:42+0100 [] INFO: Package Manager: aptitude
Daniel,
It may help to change the logging level to DEBUG in opscenterd.conf. I'm not sure what is causing the problem here. The most likely candidate I would say is a firewall issue preventing OpsCenter from connecting to port 9160 on your cluster. That doesn't explain why it would break when upgrading from 1.0.5 to 1.0.6 though.
Can you check if the debug log provides any more information.
Anonymous
Unregistered
Ok, will do so, but on my nodes there shouldnt be a firewall configured. BTW: what i recognized as well (but this is then not opscenter specific) is, that the hector client is also not connecting anymore in a client i wrote:
214 [main] INFO me.prettyprint.cassandra.connection.CassandraHostRetryService - Host detected as down was added to retry queue: xx.xx.xx.xx(xx.xx.xx.xx):9160
219 [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] WARN me.prettyprint.cassandra.connection.CassandraHostRetryService - Downed xxx.xx.xx.xx(xx.xx.xx.xx):9160 host still appears to be down: Unable to open transport to 78.46.
This is weird, because this also worked before with 1.0.5. Is opscenter also using hector?
wei
OpsCenter does not use hector, it uses another thrift client called Telephus. It sounds like thrift on your cluster is broken. Perhaps there are errors in the cassandra log?
Anonymous
Unregistered
OK, I've found it. The rpc_address-setting was set to localhost and didn't worked somehow. But it worked before. Don't know why. I have set it now to the specific IP-addresses of the nodes.
Thanks for support.
You must log in to post.
