Support Forums : DataStax

DataStax Support Forums » DataStax Enterprise

How to get Brisk AMI working for a whole cluster of nodes?

(6 posts) (2 voices)

Started 1 year ago by anthonyu
Latest reply from joaquin
This topic is not a support question

Tags:

anthonyu
Member

I cannot seem to be able to start a cluster where all nodes make it to the Up state. Even with only six nodes, at least one hangs forever when trying to log in:

Cluster started with these options:
-s 6 -c 2 -n pipeline

Waiting for nodetool...
(The cluster is now booting up. This should only take a moment.)

^CTraceback (most recent call last):
File ".motd.py", line 60, in <module>
retcode = subprocess.call(shlex.split(nodetoolStatement), stderr=subprocess.PIPE, stdout=subprocess.PIPE)
File "/usr/lib/python2.6/subprocess.py", line 470, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.6/subprocess.py", line 1182, in wait
pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
File "/usr/lib/python2.6/subprocess.py", line 455, in _eintr_retry_call
return func(*args)
KeyboardInterrupt

I hit ctrl-c and broke out of the eternal wait which is caused by receiving this exception in a while loop:

ubuntu@ip-10-251-135-80:~$ brisk/resources/cassandra/bin/nodetool -h localhost ring
Error connection to remote JMX agent!
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:141)
at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:111)
at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:559)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
... 9 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.<init>(Socket.java:375)
at java.net.Socket.<init>(Socket.java:189)
at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
... 14 more

I am using ami-da22dab3 in EC2 zone us-east-1b.

Is Brisk or this AMI just not yet ready for prime-time? Anyone have any suggestions?

Posted 1 year ago #
joaquin
Administrator

If you could upload the ~/.startlog or send me an email at joaquin at datastax.com, I could further diagnose the problem.

Looking over your options everything seems to be in order.

Could you send me the tail end of your /var/log/cassandra/cassandra.log and /var/log/cassandra/system.log as well?

Thanks,
Joaquin

Posted 1 year ago #
joaquin
Administrator

Hello,

Looking at your logs everything seems to be in order for the AMI setup.

It appears as though your cluster is in order and running for 2 hours until:

ERROR [Thread-70] 2011-08-18 15:37:21,847 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-70,5,main]
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.net.Message$MessageSerializer.deserialize(Message.java:148)
at org.apache.cassandra.net.Message$MessageSerializer.deserialize(Message.java:131)
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:116)

The AMI is to help with the initial setup and launch of a Cassandra cluster and is typically a great dev and initial testing environment. The error that you are running into is hit because the memory isn't properly tuned to the current workload.

You can visit this website here: http://datastax.com/support/tuning for more information on correctly tuning for your application.

Once the memory is correctly tuned, then by simply running 'sudo service cassandra start', the Cassandra service will relaunch and you should be in working order.

Make sure to make the appropriate changes around the ring per node.

May I ask what sort of workload you were running and how many keyspaces you were using?

Thanks,
Joaquin

Posted 1 year ago #
joaquin
Administrator

If you can try this setting as well:

sudo bash -c "echo 1 > /proc/sys/vm/overcommit_memory"

this will also help and alleviate some pressure. It should already be in the AMI codebase, but will be updated soon so that way the change can be permanent as well.

Thanks,
Joaquin

Posted 1 year ago #
anthonyu
Member

There was no workload, this was a freshly booted AMI on a cluster of m1.large instances that had never had any data loaded.

Posted 1 year ago #
joaquin
Administrator

From your logs I'm seeing your first line as:

INFO [main] 2011-08-18 03:15:08,141 AbstractCassandraDaemon.java (line 78) Logging initialized

and your last lines as:

INFO [HintedHandoff:2] 2011-08-18 05:19:32,394 HintedHandOffManager.java (line 376) Finished hinted handoff of 0 rows to endpoint /10.250.79.112
ERROR [Thread-70] 2011-08-18 15:37:21,847 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-70,5,main]
java.lang.OutOfMemoryError: Java heap space

Which lets me know that this node had been running for some time.

I'm also noticing that compacted SSTable files were removed meaning that data was already on this cluster. Did you perhaps send me a different log file?

Posted 1 year ago #

RSS feed for this topic

Reply

You must log in to post.