DataStax Support Forums » User Favorites: dmarlow

DataStax Support Forums » User Favorites: dmarlow dmarlow Software, Support, and Training for Apache Cassandra en-US Thu, 20 Jun 2013 03:06:34 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php nickmbailey on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-9012 Tue, 12 Feb 2013 18:53:01 +0000 nickmbailey 9012@http://www.datastax.com/support-forums/ You would want to stop the cassandra process on those nodes and then run 'nodetool removetoken' on one of the windows nodes to remove both of the linux nodes from the cluster. dmarlow on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-9011 Tue, 12 Feb 2013 18:46:13 +0000 dmarlow 9011@http://www.datastax.com/support-forums/ Thanks for root causing it. Any way to properly decommission/remove the nodes with agents not properly communicating? nickmbailey on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-9010 Tue, 12 Feb 2013 18:17:45 +0000 nickmbailey 9010@http://www.datastax.com/support-forums/ I believe the nodes failed to bootstrap due to the problem described here <a href="https://issues.apache.org/jira/browse/CASSANDRA-2887" rel="nofollow">https://issues.apache.org/jira/browse/CASSANDRA-2887</a> The assertion error is probably still a bug and I will file a new ticket for it but like I said earlier, mixed OS clusters aren't generally recommended for these reasons. dmarlow on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-9009 Tue, 12 Feb 2013 17:45:48 +0000 dmarlow 9009@http://www.datastax.com/support-forums/ 113 and 114 are the linux nodes. 110 and 111 (windows) have been working great. nickmbailey on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-9008 Tue, 12 Feb 2013 17:14:40 +0000 nickmbailey 9008@http://www.datastax.com/support-forums/ Or are 113 and 114 the linux nodes? There may be a cassandra bug with attempting to get the list of tokens from a bootstrapping node. nickmbailey on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-9007 Tue, 12 Feb 2013 17:10:46 +0000 nickmbailey 9007@http://www.datastax.com/support-forums/ Nodes 113 and 114 appear to still be boostrapping which seems to be causing the issue. Can you run the same commands on those nodes? Does the log indicate any errors or a successful bootstrap? dmarlow on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-8998 Mon, 11 Feb 2013 23:39:02 +0000 dmarlow 8998@http://www.datastax.com/support-forums/ Data below is the same on both linux machines. nodetool ring Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: datacenter1 ========== Address Rack Status State Load Owns Token 85070591730234615865843651857942052864 --.---.--.110 rack1 Up Normal 1.01 GB 50.00% 0 --.---.--.111 rack1 Up Normal 1.21 GB 50.00% 85070591730234615865843651857942052864 nodetool info Exception in thread "main" java.lang.AssertionError at org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465) at org.apache.cassandra.service.StorageService.getTokens(StorageService.java:1993) at org.apache.cassandra.service.StorageService.getTokens(StorageService.java:1982) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226) at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83) at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:205) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:683) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:672) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1452) at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1316) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408) at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:650) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) nodetool gossipinfo /--.---.--.111 RELEASE_VERSION:1.2.0 SEVERITY:0.0 HOST_ID:85eb9d7e-4f61-456f-946d-4b68c3ef9158 SCHEMA:7a030f07-7474-39de-a25e-f11b04144c46 NET_VERSION:6 STATUS:NORMAL,85070591730234615865843651857942052864 LOAD:1.304522858E9 RPC_ADDRESS:0.0.0.0 RACK:rack1 DC:datacenter1 /--.---.--.113 RELEASE_VERSION:1.2.1 HOST_ID:0aa0bdf5-31a9-4e3d-85db-8caf83159aff SCHEMA:7a030f07-7474-39de-a25e-f11b04144c46 NET_VERSION:6 STATUS:BOOT,127605887595351923798765477786913079296 RPC_ADDRESS:0.0.0.0 LOAD:1.70552967E8 RACK:rack1 DC:datacenter1 /--.---.--.114 RELEASE_VERSION:1.2.1 HOST_ID:4336ad90-162b-4040-bbb6-8e93051dbd9d SCHEMA:7a030f07-7474-39de-a25e-f11b04144c46 NET_VERSION:6 STATUS:BOOT,42535295865117307932921825928971026432 LOAD:1.98105246E8 RPC_ADDRESS:0.0.0.0 RACK:rack1 DC:datacenter1 /--.---.--.110 RELEASE_VERSION:1.2.0 SEVERITY:5.551115123125783E-17 HOST_ID:00ebef5f-f9e8-4b4b-affc-043e8abd3936 SCHEMA:7a030f07-7474-39de-a25e-f11b04144c46 NET_VERSION:6 STATUS:NORMAL,0 LOAD:1.081732428E9 RPC_ADDRESS:0.0.0.0 RACK:rack1 DC:datacenter1 nickmbailey on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-8997 Mon, 11 Feb 2013 22:28:47 +0000 nickmbailey 8997@http://www.datastax.com/support-forums/ Well i'll first note that running that running a mixed OS cluster is not advised at all. I'm not sure if that is somehow causing the issue here but there are likely to be other issues in cassandra even if that is not causing the issue here. In order to debug the issue further though, can you run: nodetool ring nodetool info nodetool gossipinfo on both of the ubuntu nodes? dmarlow on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-8996 Mon, 11 Feb 2013 22:07:44 +0000 dmarlow 8996@http://www.datastax.com/support-forums/ 1. How did you install cassandra? Installed using apt-get (I did a apt-get install dcs12) 2. What version of 1.2 are you running? 1.2.0/1.2.1/checkout? I believe I have the two windows nodes on 1.2.0 and the ubuntu ones on 1.2.1. 3. Are you using vnodes? (multiple tokens per host)? Not that I know of. Single token per host. using these: Node #1: 0 Node #2: 42535295865117307932921825928971026432 Node #3: 85070591730234615865843651857942052864 Node #4: 127605887595351923798765477786913079296 4. Are your nodes configured to use a separate broadcast address from listen address? Nope, broadcast is commented out. Should be using same as listen. nickmbailey on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-8993 Mon, 11 Feb 2013 18:26:01 +0000 nickmbailey 8993@http://www.datastax.com/support-forums/ That stack trace is actually coming from cassandra itself not from the agent. There may be a bug in cassandra. A couple of questions: 1. How did you install cassandra? 2. What version of 1.2 are you running? 1.2.0/1.2.1/checkout? 3. Are you using vnodes? (multiple tokens per host)? 4. Are your nodes configured to use a separate broadcast address from listen address? dmarlow on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-8992 Mon, 11 Feb 2013 18:04:43 +0000 dmarlow 8992@http://www.datastax.com/support-forums/ Any suggestions from anyone, I'll try anything at this point. :) dmarlow on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-8982 Fri, 08 Feb 2013 23:38:42 +0000 dmarlow 8982@http://www.datastax.com/support-forums/ Here's the output of doing /nodes on opscenter.. [ { load: null, has_jna: false, devices: { saved_caches: null, commitlog: null, other: null, data: null }, task_progress: { }, node_ip: "--.---.--.113", network_interfaces: null, node_version: { }, dc: "datacenter1", node_name: null, num_procs: null, streaming: { }, token: "127605887595351923798765477786913079296", data_held: null, mode: "unknown", rpc_ip: "--.---.--.113", partitions: { saved_caches: null, commitlog: null, other: null, data: null }, os: null, rack: "rack1", last_seen: 0 }, { load: 0.22, has_jna: false, devices: { saved_caches: "PhysicalDisk0", commitlog: "PhysicalDisk0", other: [ ], data: [ "PhysicalDisk0" ] }, task_progress: { }, node_ip: "--.---.--.111", network_interfaces: [ "6TO4_Adapter", "isatap__ADF14D0B_B7FD_45CA_8282_956AB380AAD4_", "Local_Area_Connection__11", "Microsoft_Hyper_V_Network_Adapter" ], node_version: { cassandra: "1.2.0" }, dc: "datacenter1", node_name: "Cassandra2", num_procs: 4, streaming: { }, token: "85070591730234615865843651857942052864", data_held: 1057183308, mode: "normal", rpc_ip: "--.---.--.111", partitions: { saved_caches: "C:", commitlog: "C:", other: [ ], data: [ "C:" ] }, os: "windows nt (unknown)", rack: "rack1", last_seen: 0 }, { load: 0.28, has_jna: false, devices: { saved_caches: "PhysicalDisk0", commitlog: "PhysicalDisk0", other: [ ], data: [ "PhysicalDisk0" ] }, task_progress: { }, node_ip: "--.---.--.110", network_interfaces: [ "6TO4_Adapter", "isatap__ADF14D0B_B7FD_45CA_8282_956AB380AAD4_", "Local_Area_Connection__11", "Microsoft_Hyper_V_Network_Adapter" ], node_version: { cassandra: "1.2.0" }, dc: "datacenter1", node_name: "Cassandra1", num_procs: 4, streaming: { }, token: "0", data_held: 873216255, mode: "normal", rpc_ip: "--.---.--.110", partitions: { saved_caches: "C:", commitlog: "C:", other: [ ], data: [ "C:" ] }, os: "windows nt (unknown)", rack: "rack1", last_seen: 0 }, { load: null, has_jna: false, devices: { saved_caches: null, commitlog: null, other: null, data: null }, task_progress: { }, node_ip: "--.---.--.114", network_interfaces: null, node_version: { }, dc: "datacenter1", node_name: null, num_procs: null, streaming: { }, token: "42535295865117307932921825928971026432", data_held: null, mode: "unknown", rpc_ip: "--.---.--.114", partitions: { saved_caches: null, commitlog: null, other: null, data: null }, os: null, rack: "rack1", last_seen: 0 } ] dmarlow on "Agents not connecting" http://www.datastax.com/support-forums/topic/agents-not-connecting#post-8981 Fri, 08 Feb 2013 23:33:38 +0000 dmarlow 8981@http://www.datastax.com/support-forums/ Currently running a 4 node topology, 2 windows (x.x.x.110 and .111) + 2 ubuntu (.113 and .114) All on cassandra 1.2, opscenter 2.1.3, agents 2.11 OpsCenter is on one of the windows boxes and the other connects just fine. The two agents on ubuntu don't connect. I'm getting the following exception when starting the agents on ubuntu: log4j:WARN No appenders could be found for logger (org.eclipse.jetty.util.log). log4j:WARN Please initialize the log4j system properly. log4j:WARN See <a href="http://logging.apache.org/log4j/1.2/faq.html#noconfig" rel="nofollow">http://logging.apache.org/log4j/1.2/faq.html#noconfig</a> for more info. INFO [main] 2013-02-08 15:23:27,015 Loading conf files: ./conf/address.yaml INFO [main] 2013-02-08 15:23:27,059 Java vendor/version: OpenJDK 64-Bit Server VM/1.6.0_24 DEBUG [main] 2013-02-08 15:23:27,076 Default config values: {:my_channel_prefix "/agent", :poll_period 60, :thrift_conn_timeout 10000, :stomp_port 61620, :shorttime_interval 10, :longtime_interval 300, :async_retry_timeout 5, :jmx_host "127.0.0.1", :metrics_enabled 1, :async_queue_size 5000, :rollup_snapshot_period 300, :monitor_command "/usr/share/opscent-agent/bin/opscenter_agent_monitor", :jmx_port 7199, :jmx_metrics_threadpool_size 4, :use_ssl 0, :nodedetails_threadpool_size 3, :api_port 61621, :jmx_thread_pool_size 5, :production 1, :stomp_interface "--.---.--.110", :rollup_snapshot_threshold 300, :thrift_max_conns 1, :realtime_interval 5} INFO [main] 2013-02-08 15:23:27,076 Waiting for the config from OpsCenter INFO [main] 2013-02-08 15:23:27,077 SSL communication is disabled INFO [main] 2013-02-08 15:23:27,077 Creating stomp connection to --.---.--.110:61620 INFO [main] 2013-02-08 15:23:27,080 Attempting to determine Cassandra's broadcast address through JMX INFO [StompConnection receiver] 2013-02-08 15:23:27,081 Reconnecting in 0s. INFO [StompConnection receiver] 2013-02-08 15:23:27,094 Connected to --.---.--.110:61620 DEBUG [StompConnection receiver] 2013-02-08 15:23:27,095 Sending request for basic conf INFO [Initialization] 2013-02-08 15:23:27,106 Sleeping for 2s before trying to determine IP over JMX again DEBUG [pool-2-thread-1] 2013-02-08 15:23:27,124 Sending message '[null,["2.11","\/1248753804\/conf",["fe80:0:0:0:a00:27ff:fed3:eed2%2","0:0:0:0:0:0:0:1%1","--.---.--.114","127.0.0.1","127.0.1.1"]]]' to channel '/conf-request'. DEBUG [pool-2-thread-2] 2013-02-08 15:23:27,131 Sending message '[null,["2.11","\/1248753804\/conf",["fe80:0:0:0:a00:27ff:fed3:eed2%2","0:0:0:0:0:0:0:1%1","--.---.--.114","127.0.0.1","127.0.1.1"]]]' to channel '/conf-request'. DEBUG [StompConnection receiver] 2013-02-08 15:23:27,150 Received stomp message: {message-id=/1248753804/conf_1, destination=/1248753804/conf} INFO [StompConnection receiver] 2013-02-08 15:23:27,160 Got new config from OpsCenter: {:rollups300_ttl 2419200, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :metrics_ignored_column_families "", :jmx_port 7199, :rollups86400_ttl -1, :api_port "61621", :metrics_ignored_solr_cores "", :metrics_ignored_keyspaces "system, system_traces, system_auth, OpsCenter"} DEBUG [StompConnection receiver] 2013-02-08 15:23:27,161 Received stomp message: {message-id=/1248753804/conf_2, destination=/1248753804/conf} INFO [StompConnection receiver] 2013-02-08 15:23:27,169 Got new config from OpsCenter: {:rollups300_ttl 2419200, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :metrics_ignored_column_families "", :jmx_port 7199, :rollups86400_ttl -1, :api_port "61621", :metrics_ignored_solr_cores "", :metrics_ignored_keyspaces "system, system_traces, system_auth, OpsCenter"} INFO [Initialization] 2013-02-08 15:23:29,111 New JMX connection (127.0.0.1:7199) Exception in thread "Initialization" java.lang.RuntimeException: javax.management.RuntimeErrorException: java.lang.AssertionError at clojure.lang.AFn.run(AFn.java:28) at java.lang.Thread.run(Thread.java:679) Caused by: javax.management.RuntimeErrorException: java.lang.AssertionError at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:879) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:747) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:699) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1456) at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1316) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408) at javax.management.remote.rmi.RMIConnectionImpl.getAttributes(RMIConnectionImpl.java:686) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:273) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:251) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttributes(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttributes(RMIConnector.java:910) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:90) at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) at clojure.java.jmx$raw_read.invoke(jmx.clj:203) at clojure.core$comp$fn__3646.invoke(core.clj:1947) at opsagent.jmx$get_attributes.invoke(jmx.clj:73) at opsagent.jmx$get_attribute.invoke(jmx.clj:84) at opsagent.jmx$get_token.invoke(jmx.clj:131) at opsagent.jmx$determine_ip$fn__1086.invoke(jmx.clj:146) at opsagent.jmx$jmx_wrap.doInvoke(jmx.clj:102) at clojure.lang.RestFn.invoke(RestFn.java:410) at opsagent.jmx$determine_ip.invoke(jmx.clj:145) at opsagent.jmx$setup_jmx$fn__1100.invoke(jmx.clj:175) at clojure.lang.AFn.run(AFn.java:24) ... 1 more Caused by: java.lang.AssertionError at org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465) at org.apache.cassandra.service.StorageService.getTokens(StorageService.java:1993) at org.apache.cassandra.service.StorageService.getTokens(StorageService.java:1982) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226) at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83) at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:205) at com.sun.jmx.mbeanserver.MBeanSupport.getAttributes(MBeanSupport.java:212) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:745) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:699) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1456) at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1316) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408) at javax.management.remote.rmi.RMIConnectionImpl.getAttributes(RMIConnectionImpl.java:686) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) ... 1 more I've disabled SSL, enabled debugging, firewalls are off on the ubuntu machines. On the opscenterd.log on the windows box with opscenter, I get the following: WARN: Unable to collect datacenter, rack information: Failed to query to <a href="http://--.---.--.114:61621/cluster/datacenter?node_ip=--.---.--.110" rel="nofollow">http://--.---.--.114:61621/cluster/datacenter?node_ip=--.---.--.110</a> : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it.. Tail end of agent log is: INFO [Initialization] 2013-02-08 15:30:16,189 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2013-02-08 15:30:36,191 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2013-02-08 15:30:56,193 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2013-02-08 15:31:16,194 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2013-02-08 15:31:36,196 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2013-02-08 15:31:56,198 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2013-02-08 15:32:16,200 Sleeping for 20s before trying to determine IP over JMX again INFO [Initialization] 2013-02-08 15:32:36,202 Sleeping for 20s before trying to determine IP over JMX again I've read just about everything out there about getting agents connected and I can't get it working. Please advise.