DataStax Support Forums » Topic: Error in OpsCenter

DataStax Support Forums » Topic: Error in OpsCenter http://www.datastax.com/support-forums/topic/error-in-opscenter Software, Support, and Training for Apache Cassandra en-US Fri, 24 May 2013 10:55:19 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php nickmbailey on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-302 Thu, 14 Jul 2011 18:13:05 +0000 nickmbailey 302@http://www.datastax.com/support-forums/ Good to hear it all got figured out. Glad you like the newest version :). blueplastic on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-301 Thu, 14 Jul 2011 17:56:15 +0000 blueplastic 301@http://www.datastax.com/support-forums/ Yup, you're right Nick. Thanks for the help, don't think I would have figured this out on my own. The brisk_system keyspace was automatically set up to place one replica in the Brisk data center (which doesn't exist since I'm not using BriskSnitch): [default@brisk_system] describe keyspace; Keyspace: brisk_system: Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy Durable Writes: true Options: [Brisk:1, Cassandra:0] I ran this command to update the keyspace: [default@brisk_system] update keyspace brisk_system with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options=[{DC2:1, DC1:0}]; 5a8ec1c0-ae40-11e0-0000-8b19d956fdfe Waiting for schema agreement... ... schemas agree across the cluster And finally: [default@brisk_system] describe keyspace; Keyspace: brisk_system: Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy Durable Writes: true Options: [DC2:1, DC1:0] OpsCenter works now and I just auto-installed the agent on the rest of the nodes. By the way, OpsCenter v1.2 is pretty sweet. nickmbailey on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-300 Thu, 14 Jul 2011 02:48:02 +0000 nickmbailey 300@http://www.datastax.com/support-forums/ So by default the brisk specific column families are created with NetworkTopologyStrategy and set to only exist in the 'Brisk' datacenter. When you use the BriskSnitch that datacenter gets set up automatically. Since you are using the property file snitch, and have no nodes in the 'Brisk' datacenter, you are seeing those errors. You can either modify the brisk column families to replicate to 'DC1' or 'DC2', or just change the name of one of those datacenters. Make sense? blueplastic on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-299 Thu, 14 Jul 2011 00:06:25 +0000 blueplastic 299@http://www.datastax.com/support-forums/ Hmm, you were right Nick. So, I started doing some investigation, but can't find the root cause of the problem here. The Cassandra system.log is littered with these errors, like you predicted: <pre><code>ERROR [pool-2-thread-10274] 2011-07-12 23:43:40,320 Cassandra.java (line 3668) Internal error processing describe_ring java.lang.IllegalStateException: datacenter (Brisk) has no more endpoints, (1) replicas still needed at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:124) at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:100) at org.apache.cassandra.service.StorageService.constructRangeToEndpointMap(StorageService.java:618) at org.apache.cassandra.service.StorageService.getRangeToAddressMap(StorageService.java:593) at org.apache.cassandra.service.StorageService.getRangeToEndpointMap(StorageService.java:567) at org.apache.cassandra.thrift.CassandraServer.describe_ring(CassandraServer.java:714) at org.apache.cassandra.thrift.Cassandra$Processor$describe_ring.process(Cassandra.java:3664) at org.apache.cassandra.thrift.Brisk$Processor.process(Brisk.java:464) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)</code></pre> So, it sounds like the brisk DC needs another replica? I don't fully understand this. I remember we got rid of this error a couple of weeks ago by clearing out /var/lib/cassandra, but we can't do that now b/c there is data on this cluster. Here is what nodetool ring shows (note, I reorganized the row output according to DC/RAC): <pre><code>Address DC Rack Status State Load Owns Token 148873535527910577765226390751398592512 10.192.143.x DC1 RAC1 Up Normal 643.42 GB 12.50% 0 10.192.171.x DC1 RAC1 Up Normal 128.96 GB 6.25% 21267647932558653966460912964485513216 10.210.95.x DC1 RAC1 Up Normal 128.34 GB 12.50% 42535295865117307932921825928971026432 10.211.19.x DC1 RAC1 Up Normal 128.55 GB 6.25% 63802943797675961899382738893456539648 10.68.58.x DC1 RAC2 Up Normal 643.05 GB 12.50% 85070591730234615865843651857942052864 10.110.31.x DC1 RAC2 Up Normal 128.84 GB 6.25% 106338239662793269832304564822427566080 10.96.58.x DC1 RAC2 Up Normal 128.11 GB 12.50% 127605887595351923798765477786913079296 10.210.195.x DC1 RAC2 Up Normal 129.33 GB 6.25% 148873535527910577765226390751398592512 10.114.138.x DC2 RAC1 Up Normal 258.04 GB 6.25% 10633823966279326983230456482242756608 10.203.79.x DC2 RAC1 Up Normal 257.14 GB 6.25% 53169119831396634916152282411213783040 10.242.209.x DC2 RAC1 Up Normal 256.58 GB 6.25% 95704415696513942849074108340184809472 10.38.25.x DC2 RAC1 Up Normal 257.08 GB 6.25% 138239711561631250781995934269155835904</code></pre> YAML file snitch settings: endpoint_snitch: org.apache.cassandra.locator.PropertyFileSnitch dynamic_snitch: true cassandra-topology.properties file: <pre><code># Cassandra Node IP=Data Center:Rack 10.192.143.x=DC1:RAC1 10.192.171.x=DC1:RAC1 10.210.95.x=DC1:RAC1 10.211.19.x=DC1:RAC1 10.68.58.x=DC1:RAC2 10.110.31.x=DC1:RAC2 10.96.58.x=DC1:RAC2 10.210.195.x=DC1:RAC2 10.114.138.x=DC2:RAC1 10.203.79.x=DC2:RAC1 10.242.209.x=DC2:RAC1 10.38.25.x=DC2:RAC1 # default for unknown nodes default=DC1:RAC1 # Native IPv6 is supported, however you must escape the colon in the IPv6 Address # Also be sure to comment out JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true" # in cassandra-env.sh #fe80\:0\:0\:0\:202\:b3ff\:fe1e\:8329=DC1:RAC3</code></pre> Our Keyspace is set to NetworkTopologyStrategy: <pre><code>[default@DemoKS] describe keyspace; Keyspace: DemoKS: Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy Durable Writes: true Options: [DC2:1, DC1:2] Column Families: ColumnFamily: DemoCF Key Validation Class: org.apache.cassandra.db.marshal.BytesType Default column value validator: org.apache.cassandra.db.marshal.BytesType Columns sorted by: org.apache.cassandra.db.marshal.BytesType Row cache size / save period in seconds: 0.0/0 Key cache size / save period in seconds: 200000.0/14400 Memtable thresholds: 1.0875/232/1440 (millions of ops/MB/minutes) GC grace seconds: 864000 Compaction min/max thresholds: 4/32 Read repair chance: 1.0 Replicate on write: false Built indexes: []</code></pre> mbulman on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-298 Wed, 13 Jul 2011 22:34:49 +0000 mbulman 298@http://www.datastax.com/support-forums/ Failure: thrift.Thrift.TApplicationException: Internal error processing describe_ring This line indicates there is some sort of problem with the nodes you're trying to connect to, most likely a schema disagreement. If you look in the cassandra log on that server, you should be able to get more info. nickmbailey on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-297 Wed, 13 Jul 2011 22:33:34 +0000 nickmbailey 297@http://www.datastax.com/support-forums/ Your problem is: Failure: thrift.Thrift.TApplicationException: Internal error processing describe_ring Unfortunately thats all the information we get back from cassandra, however if you check the cassandra logs you should see an indication of what the problem is when processing describe_ring. Often times it is a NetworkTopologyStrategy/datacenter misconfiguration. blueplastic on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-296 Wed, 13 Jul 2011 22:20:40 +0000 blueplastic 296@http://www.datastax.com/support-forums/ Thanks for the advice, Mike and Nick. So, the first node in my 12 node cluster is a seed node and the OpsCenter server. The 8th node, which is in a different logical DC is the other seed node. I logged into the 1st node/seed (the OpsCenter server) and was able to telnet into both seeds: ubuntu@domU-12-31-39-0E-x-x:~$ telnet 10.192.x.x 9160 Trying 10.192.x.x... Connected to 10.192.x.x. Escape character is '^]'. ubuntu@domU-12-31-39-0E-x-x:~$ telnet 10.114.x.x 9160 Trying 10.114.x.x... Connected to 10.114.x.x. Escape character is '^]'. Here is the output from DEBUG turned on: 2011-07-13 21:04:29+0000 [-] Log opened. 2011-07-13 21:04:29+0000 [-] twistd 10.2.0 (/usr/bin/python 2.6.6) starting up. 2011-07-13 21:04:29+0000 [-] reactor class: twisted.internet.selectreactor.SelectReactor. 2011-07-13 21:04:29+0000 [-] set uid/gid 0/0 2011-07-13 21:04:29+0000 [-] Logging level set to 'debug' 2011-07-13 21:04:29+0000 [-] OpsCenterdService startService 2011-07-13 21:04:29+0000 [-] OpsCenter version: 1.2 2011-07-13 21:04:29+0000 [-] Compatible agent version: 2.0 2011-07-13 21:04:29+0000 [-] HTTP BASIC authentication disabled 2011-07-13 21:04:29+0000 [-] agent_config items: {'jmx_user': '', 'jmx_pass': '', 'jmx_port': 7199, 'metrics_enabled': 1, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter'} 2011-07-13 21:04:29+0000 [-] opscenterd.WebServer.OpsCenterdWebServer starting on 8888 2011-07-13 21:04:29+0000 [-] Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x1876b00> 2011-07-13 21:04:29+0000 [-] Jython args: ['/usr/bin/java', '-Djava.library.path=/usr/lib/jni', '-Dpython.cachedir=/root/.jython_cache', 'org.python.util.jython', '-jar', '/usr/share/opscenter/bin/jmxjython.jar'] 2011-07-13 21:04:29+0000 [-] JmxJythonService started. State is now "startup". 2011-07-13 21:04:29+0000 [-] Starting CassandraCluster service 2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638> 2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> 2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638> 2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> 2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638> 2011-07-13 21:04:29+0000 [-] Waiting for describe_version() results 2011-07-13 21:04:29+0000 [-] morbid.morbid.StompFactory starting on 61619 2011-07-13 21:04:29+0000 [-] Starting factory <morbid.morbid.StompFactory instance at 0x1882878> 2011-07-13 21:04:29+0000 [-] morbid.morbid.StompFactory starting on 61620 2011-07-13 21:04:29+0000 [-] Starting PushService 2011-07-13 21:04:29+0000 [-] Waiting for a successful thrift call 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Thrift pool connection to <CassandraNode 10.192.x.x:9160 @0x1886638> failed Traceback (most recent call last): Failure: thrift.Thrift.TApplicationException: Internal error processing describe_ring 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.10830187797546387 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.10861492156982422 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.10861301422119141 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Thrift pool connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> failed Traceback (most recent call last): Failure: thrift.Thrift.TApplicationException: Internal error processing describe_ring 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.11836552619934082 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.10730290412902832 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.) 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.11850237846374512 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.10667991638183594 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.) 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.1185004711151123 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.097223997116088867 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.) 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.11850333213806152 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.096638917922973633 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.) 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.11850428581237793 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.096105098724365234 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.1 seconds.) 2011-07-13 21:04:29+0000 [-] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.015861272811889648 left on connect blackout 2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638> 2011-07-13 21:04:29+0000 [-] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.015218257904052734 left on connect blackout 2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638> 2011-07-13 21:04:29+0000 [-] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.014885425567626953 left on connect blackout 2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638> 2011-07-13 21:04:29+0000 [-] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.014558315277099609 left on connect blackout 2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638> 2011-07-13 21:04:29+0000 [-] not considering <CassandraNode 10.114.x.x:9160 @0x18866c8> for new connection; has 0.014231443405151367 left on connect blackout 2011-07-13 21:04:29+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x1886638> 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Thrift pool connection to <CassandraNode 10.192.x.x:9160 @0x1886638> failed Traceback (most recent call last): Failure: thrift.Thrift.TApplicationException: Internal error processing describe_ring 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.26951456069946289 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.26970458030700684 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.26970577239990234 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.26970577239990234 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] not considering <CassandraNode 10.192.x.x:9160 @0x1886638> for new connection; has 0.26970577239990234 left on connect blackout 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Adding connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> 2011-07-13 21:04:29+0000 [CassandraPoolParticipantClient,client] Thrift pool connection to <CassandraNode 10.114.x.x:9160 @0x18866c8> failed Traceback (most recent call last): Failure: thrift.Thrift.TApplicationException: Internal error processing describe_ring nickmbailey on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-295 Wed, 13 Jul 2011 17:40:17 +0000 nickmbailey 295@http://www.datastax.com/support-forums/ The problem is definitely with thrift. From the logs you posted it doesn't look like OpsCenter is even trying to connect to the hosts listed in the conf. The conf you posted looks normal at first glance, but perhaps there is a syntax error in there somewhere? I would recommend turning on DEBUG logging and seeing if that shows any more useful errors regarding thrift. mbulman on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-294 Wed, 13 Jul 2011 14:37:34 +0000 mbulman 294@http://www.datastax.com/support-forums/ Looks like OpsCenter can't talk to your nodes via thrift. Can you trying <code>telnet <IP> 9160</code> from the opscenter machine to the IPs listed in seed_hosts in opscenterd.conf? blueplastic on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-293 Tue, 12 Jul 2011 23:50:20 +0000 blueplastic 293@http://www.datastax.com/support-forums/ Quick update. I also tried temporarily opening all TCP ports in my EC2 Security Group (source: 0.0.0.0/0). Then I restarted OpsCenter: root@domU-12-31-39-0E-x-x:/home/ubuntu# sudo service opscenterd restart * Restarting Cassandra cluster manager opscenterd [ OK ] Finally, I refreshed the OpsCenter website but got the same error. I have removed the all TCP rule from my security group for now. Also, I want to clarify that all of my 12 Cassandra nodes are technically in the same Amazon region and two different Availability Zones. However, we’re using the Property Files snitch and simulating two different DCs and the 1st DC has two Racs. Here are all of the TCP ports that are current open in my security group: 1024 - 65535 sg-b04c3dd9 (BriskNodes) 22 (SSH) 0.0.0.0/0 80 (HTTP) 0.0.0.0/0 7000 0.0.0.0/0 7199 0.0.0.0/0 8012 0.0.0.0/0 8888 0.0.0.0/0 8983 0.0.0.0/0 9160 0.0.0.0/0 50030 0.0.0.0/0 50060 0.0.0.0/0 blueplastic on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-291 Tue, 12 Jul 2011 23:16:23 +0000 blueplastic 291@http://www.datastax.com/support-forums/ I decided to completely uninstall the older version of OpsCenter and moved to the new version released a couple of days ago. The new version of installed on just one of the seed nodes. When I load the web GUI for OpsCenter, I get this error: Error Loading OpsCenter OpsCenter is having trouble gathering basic information about your cluster. This usually means OpsCenter cannot connect to your cluster via thrift. Be sure to check your seed hosts in opscenterd.conf. ---- Error: No Cassandra connections available The opscenterd.conf file correctly has the seeds set as the private IPs for my Amazon EC2 nodes: # opscenterd.conf [jmx] # The default jmx port for Cassandra >= 0.8.0 is 7199. If you are using # Cassandra 0.7.*, the default is 8080, and you should change this to # reflect that. port = 7199 [webserver] port = 8888 interface = 0.0.0.0 [logging] # level may be TRACE, DEBUG, INFO, WARN, or ERROR level = INFO [cassandra] # a comma-separated list of places to try for a connection to your Cassandra # cluster: seed_hosts = 10.192.x.x,10.114.x.x # the API (Thrift) port on your Cassandra cluster api_port = 9160 [authentication] # if this file does not exist, there will be no password protection. Use the # set_passwd.py tool (included with OpsCenter) to set passwords. passwd_file = /etc/opscenter/.passwd [cassandra_metrics] # A list of keyspaces to *not* collect metrics for, separated by commas ignored_keyspaces = system, OpsCenter # A list of column families to *not* collect metrics for, separated # by commas. Each entry should be of the form "ks.cf". For example: # metrics_ignored_column_families = system.NodeInfo, system.Schema, Keyspace1.Standard1 ignored_column_families = OpsCenter log shows: 2011-07-12 23:05:09+0000 [-] Log opened. 2011-07-12 23:05:09+0000 [-] twistd 10.2.0 (/usr/bin/python 2.6.6) starting up. 2011-07-12 23:05:09+0000 [-] reactor class: twisted.internet.selectreactor.SelectReactor. 2011-07-12 23:05:09+0000 [-] set uid/gid 0/0 2011-07-12 23:05:09+0000 [-] Logging level set to 'info' 2011-07-12 23:05:09+0000 [-] OpsCenterdService startService 2011-07-12 23:05:09+0000 [-] OpsCenter version: 1.2 2011-07-12 23:05:09+0000 [-] Compatible agent version: 2.0 2011-07-12 23:05:09+0000 [-] HTTP BASIC authentication disabled 2011-07-12 23:05:09+0000 [-] agent_config items: {'jmx_user': '', 'jmx_pass': '', 'jmx_port': 7199, 'metrics_enabled': 1, 'metrics_ignored_column_families': '', 'metrics_ignored_keyspaces': 'system, OpsCenter'} 2011-07-12 23:05:09+0000 [-] opscenterd.WebServer.OpsCenterdWebServer starting on 8888 2011-07-12 23:05:09+0000 [-] Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x308cb00> 2011-07-12 23:05:09+0000 [-] JmxJythonService started. State is now "startup". 2011-07-12 23:05:09+0000 [-] Starting CassandraCluster service 2011-07-12 23:05:09+0000 [-] morbid.morbid.StompFactory starting on 61619 2011-07-12 23:05:09+0000 [-] Starting factory <morbid.morbid.StompFactory instance at 0x3098878> 2011-07-12 23:05:09+0000 [-] morbid.morbid.StompFactory starting on 61620 2011-07-12 23:05:09+0000 [-] Starting PushService 2011-07-12 23:05:09+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.) 2011-07-12 23:05:09+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.) 2011-07-12 23:05:09+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 1 pending requests. Expected candidate node retry in 0.2 seconds.) [there's tons of lines like the one above, which I removed] 2011-07-12 23:05:16+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 6.8 seconds.) 2011-07-12 23:05:16+0000 [-] JmxJythonService got jmxjython startup message. State is now "normal". 2011-07-12 23:05:22+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 3.6 seconds.) 2011-07-12 23:09:39+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 160.6 seconds.) 2011-07-12 23:09:39+0000 [CassandraPoolParticipantClient,client] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 160.6 seconds.) 2011-07-12 23:10:37+0000 [HTTPChannel,3,152.179.x.x] Problem while calling ClusterController Traceback (most recent call last): Failure: opscenterd.CassandraService.NoCassandraConnection: No Cassandra connections available All the ports I mentioned in the first post are still open in the EC2 security group. Would it help if I turned on higher level debugging in OpsCenter? Any ideas? joaquin on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-255 Sat, 02 Jul 2011 20:31:57 +0000 joaquin 255@http://www.datastax.com/support-forums/ Okay, well enjoy your vacation and we will definitely pick this up after the summit. And yes, OpsCenter should only be installed on one machine, preferable a separate machine. With the AMI code, I made it check the indexid was 0 before installing. You could also do something like this to see if it helps lower the overall load. blueplastic on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-254 Sat, 02 Jul 2011 00:49:34 +0000 blueplastic 254@http://www.datastax.com/support-forums/ Port change didn't fix the problem. Could be an issue related to how OpsCenter is installed in our cluster. OpsCenter was installed on the custom AMI we've created. So, when we spawn a cluster, all nodes technically have OpsCenter installed and the service running. I will try uninstalling OpsCenter from all of the nodes, except one... or at least stop the OpsCenter service on the other nodes. Also, I'm not sure what kind of problems having OpsCenter installed on all nodes would create. Does each node try to create a new OpsCenter column family? Anyway, will be on vacation for a week, so I'll have to pick back up on this issue on July 13th, after the Cassandra summit. joaquin on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-251 Fri, 01 Jul 2011 01:39:05 +0000 joaquin 251@http://www.datastax.com/support-forums/ For Brisk and Cassandra, you need to switch to port 7199 for JMX. Could you give that a try and let us know if that works? Also, OpsCenter must be started before the Cassandra cluster and restarted after every conf change. blueplastic on "Error in OpsCenter" http://www.datastax.com/support-forums/topic/error-in-opscenter#post-249 Fri, 01 Jul 2011 01:09:11 +0000 blueplastic 249@http://www.datastax.com/support-forums/ This is probably something simple. Trying to get OpsCenter to work on my 12 node Brisk-b2 cluster in Amazon. My Security Group has these ports open: 8888 9160 1024 - 65535 (source is the security group, so only traffic originating within the security group can talk via these ports) (all of the other ports mentioned in the Brisk AMI on Amazon documentation on this website are also open) OpsCenter was successfully installed on a Cassandra seed node and the cassandra-env.sh file was changed on all nodes with the updated EC2 public DNS name. Cassandra was also restarted on all nodes. OpsCenter.conf file: # opscenterd.conf [jmx] cassandra_classpath = /usr/share/opscenter/lib/*.jar port = 8080 [webserver] port = 8888 interface = 0.0.0.0 staticdir = /usr/share/opscenter/content [cassandra] forget_nodes_after = 2629800 ; average month # a comma-separated list of places to try for a connection to your Cassandra # cluster: seed_hosts = 10.192.x.x # the API (Thrift) port on your Cassandra cluster api_port = 9160 [collection] basic_info_period = 3600 node_poll_period = 120 [authentication] # if this file does not exist, there will be no password protection. Use the # set_passwd.py tool (included with OpsCenter) to set passwords. passwd_file = /etc/opscenter/.passwd - - - - - - - When I go to the URL for OpsCenter now, I see Loading Opscenter... then an error: Error loading cluster: No Cassandra connections available Here is the Opscenter log (note, IP addresses have been cleaned with x.x): 2011-07-01 00:54:55+0000 [Uninitialized] Thrift pool connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8> failed Traceback (most recent call last): Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused. 2011-07-01 00:54:55+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 180.96665978431702 left on connect blackout 2011-07-01 00:54:55+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 181.0 seconds.) 2011-07-01 00:54:55+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 180.96680474281311 left on connect blackout 2011-07-01 00:54:55+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 181.0 seconds.) 2011-07-01 00:54:55+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 180.96680688858032 left on connect blackout 2011-07-01 00:54:55+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 181.0 seconds.) 2011-07-01 00:54:55+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 180.96680688858032 left on connect blackout 2011-07-01 00:54:55+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 181.0 seconds.) 2011-07-01 00:54:55+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 180.96680569648743 left on connect blackout 2011-07-01 00:54:55+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 181.0 seconds.) 2011-07-01 00:55:32+0000 [HTTPChannel,10,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:55:32 +0000] "GET /hadoop/jobs HTTP/1.1" 404 72 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:56:32+0000 [HTTPChannel,10,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:56:32 +0000] "GET /hadoop/jobs HTTP/1.1" 404 72 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:32+0000 [HTTPChannel,10,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:32 +0000] "GET /hadoop/jobs HTTP/1.1" 404 72 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:39+0000 [HTTPChannel,10,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:39 +0000] "GET /opscenter/index.html HTTP/1.1" 200 2599 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:39+0000 [HTTPChannel,2,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:39 +0000] "GET /opscenter/js/dojotoolkit/dojox/grid/resources/Grid.css HTTP/1.1" 200 7125 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:39+0000 [HTTPChannel,9,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:39 +0000] "GET /opscenter/css/ripcord.css HTTP/1.1" 200 25573 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:39+0000 [HTTPChannel,14,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:39 +0000] "GET /opscenter/images/opsc-logo.png HTTP/1.1" 200 3052 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:40+0000 [HTTPChannel,10,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:39 +0000] "GET /opscenter/js/dojotoolkit/dijit/themes/tundra/tundra.css HTTP/1.1" 200 76329 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:40+0000 [HTTPChannel,8,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:40 +0000] "GET /opscenter/js/dojotoolkit/dojo/dojo.js HTTP/1.1" 200 90325 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:41+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:41 +0000] "GET /opscenter/js/ripcord/ripcord.js HTTP/1.1" 200 982295 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:42+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:41 +0000] "GET /opscenter/js/dojotoolkit/dojox/gfx/svg.js?1309481862161 HTTP/1.1" 200 39650 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:42+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /opscenter/js/dojotoolkit/dojo/nls/ripcorddojo_en-us.js?1309481862161 HTTP/1.1" 200 5540 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:42+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /opscenter/images/sidebar-line.png HTTP/1.1" 200 291 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:42+0000 [HTTPChannel,8,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /opscenter/images/sidebar-bg.png HTTP/1.1" 200 514 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:42+0000 [HTTPChannel,8,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /rc/userrole HTTP/1.1" 200 4 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:42+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /rc/version HTTP/1.1" 200 7 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:42+0000 [HTTPChannel,13,152.179.x.x] Problem while calling ClusterController Traceback (most recent call last): Failure: opscenterd.CassandraService.NoCassandraConnection: No Cassandra connections available 2011-07-01 00:57:42+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /cluster HTTP/1.1" 400 49 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:42+0000 [HTTPChannel,8,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /favicon.ico HTTP/1.1" 404 52 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:43+0000 [HTTPChannel,8,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /opscenter/images/closeicon.png HTTP/1.1" 200 1238 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:43+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:57:42 +0000] "GET /opscenter/js/dojotoolkit/dijit/themes/tundra/images/buttonEnabled.png HTTP/1.1" 200 120 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30" 2011-07-01 00:57:56+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8> 2011-07-01 00:57:56+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8> 2011-07-01 00:57:56+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8> 2011-07-01 00:57:56+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8> 2011-07-01 00:57:56+0000 [-] Adding connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8> 2011-07-01 00:57:56+0000 [Uninitialized] Thrift pool connection to <CassandraNode 10.192.x.x:9160 @0x2eb66c8> failed Traceback (most recent call last): Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused. 2011-07-01 00:57:56+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 174.66835308074951 left on connect blackout 2011-07-01 00:57:56+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 174.7 seconds.) 2011-07-01 00:57:56+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 174.66850137710571 left on connect blackout 2011-07-01 00:57:56+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 174.7 seconds.) 2011-07-01 00:57:56+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 174.66850113868713 left on connect blackout 2011-07-01 00:57:56+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 174.7 seconds.) 2011-07-01 00:57:56+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 174.66850018501282 left on connect blackout 2011-07-01 00:57:56+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 174.7 seconds.) 2011-07-01 00:57:56+0000 [Uninitialized] not considering <CassandraNode 10.192.x.x:9160 @0x2eb66c8> for new connection; has 174.66850018501282 left on connect blackout 2011-07-01 00:57:56+0000 [Uninitialized] (No candidate nodes to expand pool to target size 5 from 0; there are 0 pending requests. Expected candidate node retry in 174.7 seconds.) 2011-07-01 00:58:03+0000 [HTTPChannel,13,152.179.x.x] 152.179.x.x - - [01/Jul/2011:00:58:03 +0000] "GET /opscenter/js/dojotoolkit/dijit/themes/tundra/images/buttonHover.png HTTP/1.1" 200 144 "http://ec2-174-129-x-x.compute-1.amazonaws.com:8888/opscenter/index.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.x.x Safari/534.30"