Agents not connecting

(13 posts) (2 voices)

• Started 3 months ago by dmarlow
• Latest reply from nickmbailey
• This topic is not resolved

Tags:

No tags yet.

1. 

   dmarlow
   Member

   Currently running a 4 node topology, 2 windows (x.x.x.110 and .111) + 2 ubuntu (.113 and .114)
   All on cassandra 1.2, opscenter 2.1.3, agents 2.11
   OpsCenter is on one of the windows boxes and the other connects just fine. The two agents on ubuntu don't connect.

   I'm getting the following exception when starting the agents on ubuntu:

   log4j:WARN No appenders could be found for logger (org.eclipse.jetty.util.log).
   log4j:WARN Please initialize the log4j system properly.
   log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
   INFO [main] 2013-02-08 15:23:27,015 Loading conf files: ./conf/address.yaml
   INFO [main] 2013-02-08 15:23:27,059 Java vendor/version: OpenJDK 64-Bit Server VM/1.6.0_24
   DEBUG [main] 2013-02-08 15:23:27,076 Default config values: {:my_channel_prefix "/agent", :poll_period 60, :thrift_conn_timeout 10000, :stomp_port 61620, :shorttime_interval 10, :longtime_interval 300, :async_retry_timeout 5, :jmx_host "127.0.0.1", :metrics_enabled 1, :async_queue_size 5000, :rollup_snapshot_period 300, :monitor_command "/usr/share/opscent-agent/bin/opscenter_agent_monitor", :jmx_port 7199, :jmx_metrics_threadpool_size 4, :use_ssl 0, :nodedetails_threadpool_size 3, :api_port 61621, :jmx_thread_pool_size 5, :production 1, :stomp_interface "--.---.--.110", :rollup_snapshot_threshold 300, :thrift_max_conns 1, :realtime_interval 5}
   INFO [main] 2013-02-08 15:23:27,076 Waiting for the config from OpsCenter
   INFO [main] 2013-02-08 15:23:27,077 SSL communication is disabled
   INFO [main] 2013-02-08 15:23:27,077 Creating stomp connection to --.---.--.110:61620
   INFO [main] 2013-02-08 15:23:27,080 Attempting to determine Cassandra's broadcast address through JMX
   INFO [StompConnection receiver] 2013-02-08 15:23:27,081 Reconnecting in 0s.
   INFO [StompConnection receiver] 2013-02-08 15:23:27,094 Connected to --.---.--.110:61620
   DEBUG [StompConnection receiver] 2013-02-08 15:23:27,095 Sending request for basic conf
   INFO [Initialization] 2013-02-08 15:23:27,106 Sleeping for 2s before trying to determine IP over JMX again
   DEBUG [pool-2-thread-1] 2013-02-08 15:23:27,124 Sending message '[null,["2.11","\/1248753804\/conf",["fe80:0:0:0:a00:27ff:fed3:eed2%2","0:0:0:0:0:0:0:1%1","--.---.--.114","127.0.0.1","127.0.1.1"]]]' to channel '/conf-request'.
   DEBUG [pool-2-thread-2] 2013-02-08 15:23:27,131 Sending message '[null,["2.11","\/1248753804\/conf",["fe80:0:0:0:a00:27ff:fed3:eed2%2","0:0:0:0:0:0:0:1%1","--.---.--.114","127.0.0.1","127.0.1.1"]]]' to channel '/conf-request'.
   DEBUG [StompConnection receiver] 2013-02-08 15:23:27,150 Received stomp message: {message-id=/1248753804/conf_1, destination=/1248753804/conf}
   INFO [StompConnection receiver] 2013-02-08 15:23:27,160 Got new config from OpsCenter: {:rollups300_ttl 2419200, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :metrics_ignored_column_families "", :jmx_port 7199, :rollups86400_ttl -1, :api_port "61621", :metrics_ignored_solr_cores "", :metrics_ignored_keyspaces "system, system_traces, system_auth, OpsCenter"}
   DEBUG [StompConnection receiver] 2013-02-08 15:23:27,161 Received stomp message: {message-id=/1248753804/conf_2, destination=/1248753804/conf}
   INFO [StompConnection receiver] 2013-02-08 15:23:27,169 Got new config from OpsCenter: {:rollups300_ttl 2419200, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :metrics_ignored_column_families "", :jmx_port 7199, :rollups86400_ttl -1, :api_port "61621", :metrics_ignored_solr_cores "", :metrics_ignored_keyspaces "system, system_traces, system_auth, OpsCenter"}
   INFO [Initialization] 2013-02-08 15:23:29,111 New JMX connection (127.0.0.1:7199)
   Exception in thread "Initialization" java.lang.RuntimeException: javax.management.RuntimeErrorException: java.lang.AssertionError
   at clojure.lang.AFn.run(AFn.java:28)
   at java.lang.Thread.run(Thread.java:679)
   Caused by: javax.management.RuntimeErrorException: java.lang.AssertionError
   at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:879)
   at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:747)
   at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:699)
   at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1456)
   at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
   at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1316)
   at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)
   at javax.management.remote.rmi.RMIConnectionImpl.getAttributes(RMIConnectionImpl.java:686)
   at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:616)
   at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
   at sun.rmi.transport.Transport$1.run(Transport.java:177)
   at java.security.AccessController.doPrivileged(Native Method)
   at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
   at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
   at java.lang.Thread.run(Thread.java:679)
   at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:273)
   at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:251)
   at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160)
   at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
   at javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttributes(Unknown Source)
   at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttributes(RMIConnector.java:910)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:616)
   at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:90)
   at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
   at clojure.java.jmx$raw_read.invoke(jmx.clj:203)
   at clojure.core$comp$fn__3646.invoke(core.clj:1947)
   at opsagent.jmx$get_attributes.invoke(jmx.clj:73)
   at opsagent.jmx$get_attribute.invoke(jmx.clj:84)
   at opsagent.jmx$get_token.invoke(jmx.clj:131)
   at opsagent.jmx$determine_ip$fn__1086.invoke(jmx.clj:146)
   at opsagent.jmx$jmx_wrap.doInvoke(jmx.clj:102)
   at clojure.lang.RestFn.invoke(RestFn.java:410)
   at opsagent.jmx$determine_ip.invoke(jmx.clj:145)
   at opsagent.jmx$setup_jmx$fn__1100.invoke(jmx.clj:175)
   at clojure.lang.AFn.run(AFn.java:24)
   ... 1 more
   Caused by: java.lang.AssertionError
   at org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465)
   at org.apache.cassandra.service.StorageService.getTokens(StorageService.java:1993)
   at org.apache.cassandra.service.StorageService.getTokens(StorageService.java:1982)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:616)
   at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
   at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
   at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
   at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
   at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:205)
   at com.sun.jmx.mbeanserver.MBeanSupport.getAttributes(MBeanSupport.java:212)
   at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:745)
   at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:699)
   at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1456)
   at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
   at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1316)
   at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)
   at javax.management.remote.rmi.RMIConnectionImpl.getAttributes(RMIConnectionImpl.java:686)
   at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:616)
   at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
   at sun.rmi.transport.Transport$1.run(Transport.java:177)
   at java.security.AccessController.doPrivileged(Native Method)
   at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
   at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
   ... 1 more

   I've disabled SSL, enabled debugging, firewalls are off on the ubuntu machines.

   On the opscenterd.log on the windows box with opscenter, I get the following:
   WARN: Unable to collect datacenter, rack information: Failed to query to http://--.---.--.114:61621/cluster/datacenter?node_ip=--.---.--.110 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..

   Tail end of agent log is:
   INFO [Initialization] 2013-02-08 15:30:16,189 Sleeping for 20s before trying to determine IP over JMX again
   INFO [Initialization] 2013-02-08 15:30:36,191 Sleeping for 20s before trying to determine IP over JMX again
   INFO [Initialization] 2013-02-08 15:30:56,193 Sleeping for 20s before trying to determine IP over JMX again
   INFO [Initialization] 2013-02-08 15:31:16,194 Sleeping for 20s before trying to determine IP over JMX again
   INFO [Initialization] 2013-02-08 15:31:36,196 Sleeping for 20s before trying to determine IP over JMX again
   INFO [Initialization] 2013-02-08 15:31:56,198 Sleeping for 20s before trying to determine IP over JMX again
   INFO [Initialization] 2013-02-08 15:32:16,200 Sleeping for 20s before trying to determine IP over JMX again
   INFO [Initialization] 2013-02-08 15:32:36,202 Sleeping for 20s before trying to determine IP over JMX again

   I've read just about everything out there about getting agents connected and I can't get it working.

   Please advise.

   Posted 3 months ago #
2. 

   dmarlow
   Member

   Here's the output of doing /nodes on opscenter..

   [
   {
   load: null,
   has_jna: false,
   devices: {
   saved_caches: null,
   commitlog: null,
   other: null,
   data: null
   },
   task_progress: { },
   node_ip: "--.---.--.113",
   network_interfaces: null,
   node_version: { },
   dc: "datacenter1",
   node_name: null,
   num_procs: null,
   streaming: { },
   token: "127605887595351923798765477786913079296",
   data_held: null,
   mode: "unknown",
   rpc_ip: "--.---.--.113",
   partitions: {
   saved_caches: null,
   commitlog: null,
   other: null,
   data: null
   },
   os: null,
   rack: "rack1",
   last_seen: 0
   },
   {
   load: 0.22,
   has_jna: false,
   devices: {
   saved_caches: "PhysicalDisk0",
   commitlog: "PhysicalDisk0",
   other: [ ],
   data: [
   "PhysicalDisk0"
   ]
   },
   task_progress: { },
   node_ip: "--.---.--.111",
   network_interfaces: [
   "6TO4_Adapter",
   "isatap__ADF14D0B_B7FD_45CA_8282_956AB380AAD4_",
   "Local_Area_Connection__11",
   "Microsoft_Hyper_V_Network_Adapter"
   ],
   node_version: {
   cassandra: "1.2.0"
   },
   dc: "datacenter1",
   node_name: "Cassandra2",
   num_procs: 4,
   streaming: { },
   token: "85070591730234615865843651857942052864",
   data_held: 1057183308,
   mode: "normal",
   rpc_ip: "--.---.--.111",
   partitions: {
   saved_caches: "C:",
   commitlog: "C:",
   other: [ ],
   data: [
   "C:"
   ]
   },
   os: "windows nt (unknown)",
   rack: "rack1",
   last_seen: 0
   },
   {
   load: 0.28,
   has_jna: false,
   devices: {
   saved_caches: "PhysicalDisk0",
   commitlog: "PhysicalDisk0",
   other: [ ],
   data: [
   "PhysicalDisk0"
   ]
   },
   task_progress: { },
   node_ip: "--.---.--.110",
   network_interfaces: [
   "6TO4_Adapter",
   "isatap__ADF14D0B_B7FD_45CA_8282_956AB380AAD4_",
   "Local_Area_Connection__11",
   "Microsoft_Hyper_V_Network_Adapter"
   ],
   node_version: {
   cassandra: "1.2.0"
   },
   dc: "datacenter1",
   node_name: "Cassandra1",
   num_procs: 4,
   streaming: { },
   token: "0",
   data_held: 873216255,
   mode: "normal",
   rpc_ip: "--.---.--.110",
   partitions: {
   saved_caches: "C:",
   commitlog: "C:",
   other: [ ],
   data: [
   "C:"
   ]
   },
   os: "windows nt (unknown)",
   rack: "rack1",
   last_seen: 0
   },
   {
   load: null,
   has_jna: false,
   devices: {
   saved_caches: null,
   commitlog: null,
   other: null,
   data: null
   },
   task_progress: { },
   node_ip: "--.---.--.114",
   network_interfaces: null,
   node_version: { },
   dc: "datacenter1",
   node_name: null,
   num_procs: null,
   streaming: { },
   token: "42535295865117307932921825928971026432",
   data_held: null,
   mode: "unknown",
   rpc_ip: "--.---.--.114",
   partitions: {
   saved_caches: null,
   commitlog: null,
   other: null,
   data: null
   },
   os: null,
   rack: "rack1",
   last_seen: 0
   }
   ]

   Posted 3 months ago #
3. 

   dmarlow
   Member

   Any suggestions from anyone, I'll try anything at this point. :)

   Posted 3 months ago #
4. 

   nickmbailey
   Key Master

   That stack trace is actually coming from cassandra itself not from the agent. There may be a bug in cassandra. A couple of questions:

   1. How did you install cassandra?
   2. What version of 1.2 are you running? 1.2.0/1.2.1/checkout?
   3. Are you using vnodes? (multiple tokens per host)?
   4. Are your nodes configured to use a separate broadcast address from listen address?

   Posted 3 months ago #
5. 

   dmarlow
   Member

   1. How did you install cassandra?
   Installed using apt-get (I did a apt-get install dcs12)
   2. What version of 1.2 are you running? 1.2.0/1.2.1/checkout?
   I believe I have the two windows nodes on 1.2.0 and the ubuntu ones on 1.2.1.
   3. Are you using vnodes? (multiple tokens per host)?
   Not that I know of. Single token per host.
   using these:
   Node #1: 0
   Node #2: 42535295865117307932921825928971026432
   Node #3: 85070591730234615865843651857942052864
   Node #4: 127605887595351923798765477786913079296
   4. Are your nodes configured to use a separate broadcast address from listen address?
   Nope, broadcast is commented out. Should be using same as listen.

   Posted 3 months ago #
6. 

   nickmbailey
   Key Master

   Well i'll first note that running that running a mixed OS cluster is not advised at all. I'm not sure if that is somehow causing the issue here but there are likely to be other issues in cassandra even if that is not causing the issue here. In order to debug the issue further though, can you run:

   nodetool ring
   nodetool info
   nodetool gossipinfo

   on both of the ubuntu nodes?

   Posted 3 months ago #
7. 

   dmarlow
   Member

   Data below is the same on both linux machines.

   nodetool ring
   Note: Ownership information does not include topology; for complete information, specify a keyspace

   Datacenter: datacenter1
   ==========
   Address Rack Status State Load Owns Token
   85070591730234615865843651857942052864
   --.---.--.110 rack1 Up Normal 1.01 GB 50.00% 0
   --.---.--.111 rack1 Up Normal 1.21 GB 50.00% 85070591730234615865843651857942052864

   nodetool info
   Exception in thread "main" java.lang.AssertionError
   at org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465)
   at org.apache.cassandra.service.StorageService.getTokens(StorageService.java:1993)
   at org.apache.cassandra.service.StorageService.getTokens(StorageService.java:1982)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:616)
   at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
   at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
   at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
   at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
   at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:205)
   at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:683)
   at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:672)
   at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1452)
   at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
   at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1316)
   at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)
   at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:650)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:616)
   at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
   at sun.rmi.transport.Transport$1.run(Transport.java:177)
   at java.security.AccessController.doPrivileged(Native Method)
   at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
   at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
   at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
   at java.lang.Thread.run(Thread.java:679)

   nodetool gossipinfo
   /--.---.--.111
   RELEASE_VERSION:1.2.0
   SEVERITY:0.0
   HOST_ID:85eb9d7e-4f61-456f-946d-4b68c3ef9158
   SCHEMA:7a030f07-7474-39de-a25e-f11b04144c46
   NET_VERSION:6
   STATUS:NORMAL,85070591730234615865843651857942052864
   LOAD:1.304522858E9
   RPC_ADDRESS:0.0.0.0
   RACK:rack1
   DC:datacenter1
   /--.---.--.113
   RELEASE_VERSION:1.2.1
   HOST_ID:0aa0bdf5-31a9-4e3d-85db-8caf83159aff
   SCHEMA:7a030f07-7474-39de-a25e-f11b04144c46
   NET_VERSION:6
   STATUS:BOOT,127605887595351923798765477786913079296
   RPC_ADDRESS:0.0.0.0
   LOAD:1.70552967E8
   RACK:rack1
   DC:datacenter1
   /--.---.--.114
   RELEASE_VERSION:1.2.1
   HOST_ID:4336ad90-162b-4040-bbb6-8e93051dbd9d
   SCHEMA:7a030f07-7474-39de-a25e-f11b04144c46
   NET_VERSION:6
   STATUS:BOOT,42535295865117307932921825928971026432
   LOAD:1.98105246E8
   RPC_ADDRESS:0.0.0.0
   RACK:rack1
   DC:datacenter1
   /--.---.--.110
   RELEASE_VERSION:1.2.0
   SEVERITY:5.551115123125783E-17
   HOST_ID:00ebef5f-f9e8-4b4b-affc-043e8abd3936
   SCHEMA:7a030f07-7474-39de-a25e-f11b04144c46
   NET_VERSION:6
   STATUS:NORMAL,0
   LOAD:1.081732428E9
   RPC_ADDRESS:0.0.0.0
   RACK:rack1
   DC:datacenter1

   Posted 3 months ago #
8. 

   nickmbailey
   Key Master

   Nodes 113 and 114 appear to still be boostrapping which seems to be causing the issue. Can you run the same commands on those nodes? Does the log indicate any errors or a successful bootstrap?

   Posted 3 months ago #
9. 

   nickmbailey
   Key Master

   Or are 113 and 114 the linux nodes? There may be a cassandra bug with attempting to get the list of tokens from a bootstrapping node.

   Posted 3 months ago #
10. 

    dmarlow
    Member

    113 and 114 are the linux nodes.

    110 and 111 (windows) have been working great.

    Posted 3 months ago #
11. 

    nickmbailey
    Key Master

    I believe the nodes failed to bootstrap due to the problem described here https://issues.apache.org/jira/browse/CASSANDRA-2887

    The assertion error is probably still a bug and I will file a new ticket for it but like I said earlier, mixed OS clusters aren't generally recommended for these reasons.

    Posted 3 months ago #
12. 

    dmarlow
    Member

    Thanks for root causing it. Any way to properly decommission/remove the nodes with agents not properly communicating?

    Posted 3 months ago #
13. 

    nickmbailey
    Key Master

    You would want to stop the cassandra process on those nodes and then run 'nodetool removetoken' on one of the windows nodes to remove both of the linux nodes from the cluster.

    Posted 3 months ago #

RSS feed for this topic

Reply

You must log in to post.