DataStax Support Forums » Topic: Node went Down in Cassandra cluster

DataStax Support Forums » Topic: Node went Down in Cassandra cluster http://www.datastax.com/support-forums/topic/node-went-down-in-cassandra-cluster Software, Support, and Training for Apache Cassandra en-US Sun, 19 May 2013 05:43:22 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php Anonymous on "Node went Down in Cassandra cluster" http://www.datastax.com/support-forums/topic/node-went-down-in-cassandra-cluster#post-1086 Fri, 03 Feb 2012 19:05:51 +0000 Anonymous 1086@http://www.datastax.com/support-forums/ There is no Error messages in the Cassandra logs, just the Info messages, what do I need change so that I could see the Errors as well in the logs? thepaul on "Node went Down in Cassandra cluster" http://www.datastax.com/support-forums/topic/node-went-down-in-cassandra-cluster#post-1084 Thu, 02 Feb 2012 16:56:37 +0000 thepaul 1084@http://www.datastax.com/support-forums/ That just means nodetool couldn't make a TCP connection to the cassandra instance on the dead node (probably because it's dead). You'll probably find more useful information in the cassandra logs. Anonymous on "Node went Down in Cassandra cluster" http://www.datastax.com/support-forums/topic/node-went-down-in-cassandra-cluster#post-1080 Thu, 02 Feb 2012 00:07:32 +0000 Anonymous 1080@http://www.datastax.com/support-forums/ Hi Experts, one of our nodes went down unexpectedly. I am trying to investigate what might be root cause for this happen. I did a <code>nodetool -h localhost info</code> on the dead node & got this Error connection to remote JMX agent! java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: java.net.ConnectException: Connection refused] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:340) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248) at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:143) at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:113) at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:589) Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: java.net.ConnectException: Connection refused] at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101) at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185) at javax.naming.InitialContext.lookup(InitialContext.java:392) at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1888) at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1858) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257) ... 4 more Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: java.net.ConnectException: Connection refused at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184) at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322) at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97) ... 9 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at java.net.Socket.<init>(Socket.java:375) at java.net.Socket.<init>(Socket.java:189) at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22) at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595) ... 14 more Could you please throw me some hints. Thanks, Nish