Support Forums : DataStax

DataStax Support Forums » Apache Cassandra

Cassandra crashes during simultaneous reads and writes on EC2

(4 posts) (2 voices)

Started 1 year ago by
Latest reply from jbellis
This topic is not resolved

Tags:

No tags yet.

Anonymous
Unregistered

Hi,

We are currently using a cluster of 2 cassandra nodes on 2 larges EC2 instances. Each of them has about 65G of used data. ( Same setup for Dev and Production ).

The issue we are having is during writes on production - the CPU load becomes 40 + ( 2 core machine ) and eventually the machines becomes unavailable and need to be rebooted. I tried mutiple "tunning strategies" such as decreasing the total memtable space, changing ratio of eden space vs survival space in the young generation, copying larger object directly from Eden space to the Older Generation, optimizing compaction to run more frequently, using smaller # of sstables ( I noticed that during the crash disk utilization was almost none, so I tried to relieve the memory usage).

Looking at cassandra system.log, I wasn't able to see any ERRORS or WARNINGs.
Only thing that shows up during the crash is the StatusLogger :

INFO [ScheduledTasks:1] 2012-03-08 08:21:21,741 StatusLogger.java (line 50) Pool Name INFO [ScheduledTasks:1] 2012-03-08 08:21:21,741 StatusLogger.java (line 65) ReadStage INFO [ScheduledTasks:1] 2012-03-08 08:21:21,742 StatusLogger.java (line 65) RequestResponseStage INFO [ScheduledTasks:1] 2012-03-08 08:21:21,742 StatusLogger.java (line 65) ReadRepairStage INFO [ScheduledTasks:1] 2012-03-08 08:21:21,742 StatusLogger.java (line 65) MutationStage INFO [ScheduledTasks:1] 2012-03-08 08:21:21,743 StatusLogger.java (line 65) ReplicateOnWriteStage INFO [ScheduledTasks:1] 2012-03-08 08:21:21,743 StatusLogger.java (line 65) GossipStage INFO [ScheduledTasks:1] 2012-03-08 08:21:21,744 StatusLogger.java (line 65) AntiEntropyStage INFO [ScheduledTasks:1] 2012-03-08 08:21:21,744 StatusLogger.java (line 65) MigrationStage INFO [ScheduledTasks:1] 2012-03-08 08:21:21,744 StatusLogger.java (line 65) StreamStage INFO [ScheduledTasks:1] 2012-03-08 08:21:21,744 StatusLogger.java (line 65) MemtablePostFlusher INFO [ScheduledTasks:1] 2012-03-08 08:21:21,745 StatusLogger.java (line 65) FlushWriter INFO [ScheduledTasks:1] 2012-03-08 08:21:21,760 StatusLogger.java (line 65) MiscStage INFO [ScheduledTasks:1] 2012-03-08 08:21:21,760 StatusLogger.java (line 65) InternalResponseStage INFO [ScheduledTasks:1] 2012-03-08 08:21:21,761 StatusLogger.java (line 65) HintedHandoff INFO [ScheduledTasks:1] 2012-03-08 08:21:21,761 StatusLogger.java (line 69) CompactionManager INFO [ScheduledTasks:1] 2012-03-08 08:21:21,761 StatusLogger.java (line 81) MessagingService INFO [ScheduledTasks:1] 2012-03-08 08:21:21,761 StatusLogger.java (line 85) ColumnFamily INFO [ScheduledTasks:1] 2012-03-08 08:21:21,762 StatusLogger.java (line 88) system.NodeIdInfo INFO [ScheduledTasks:1] 2012-03-08 08:21:21,762 StatusLogger.java (line 88) system.IndexInfo INFO [ScheduledTasks:1] 2012-03-08 08:21:21,762 StatusLogger.java (line 88) system.LocationInfo INFO [ScheduledTasks:1] 2012-03-08 08:21:21,762 StatusLogger.java (line 88) system.Versions INFO [ScheduledTasks:1] 2012-03-08 08:21:21,763 StatusLogger.java (line 88) system.Migrations INFO [ScheduledTasks:1] 2012-03-08 08:21:21,763 StatusLogger.java (line 88) system.HintsColumnFamily INFO [ScheduledTasks:1] 2012-03-08 08:21:21,763 StatusLogger.java (line 88) system.Schema INFO [ScheduledTasks:1] 2012-03-08 08:21:21,763 StatusLogger.java (line 88) test.popular_neighbors INFO [ScheduledTasks:1] 2012-03-08 08:21:21,764 StatusLogger.java (line 88) test.popular_neighbors_root INFO [ScheduledTasks:1] 2012-03-08 08:21:21,764 StatusLogger.java (line 88) upp.topcat INFO [ScheduledTasks:1] 2012-03-08 08:21:21,764 StatusLogger.java (line 88) upp.fulllisting INFO [ScheduledTasks:1] 2012-03-08 08:21:21,764 StatusLogger.java (line 88) collector.seo_tags INFO [ScheduledTasks:1] 2012-03-08 08:21:21,765 StatusLogger.java (line 88) collector.seo_tags_full_ids INFO [ScheduledTasks:1] 2012-03-08 08:21:21,765 StatusLogger.java (line 88) collector.seo_tags_full Active Pending Blocked
1 1 0
0 0 0
0 0 0
1 600 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
0 0 0
n/a 2
n/a 0,45
Memtable ops,data Row cache size/cap Key cache size/cap
0,0 0/0 0/1
0,0 0/0 0/1
0,0 0/0 2/2
3,103 0/0 0/2
0,0 0/0 0/3
0,0 0/0 1/1
0,0 0/0 2/3
0,0 0/0 105549/200000
0,0 0/0 0/200000
113,73 0/0 31472/200000
295447,74865591 0/0 101034/200000
0,0 0/0 1104/200000
0,0 0/0 8/200000
0,0 0/0 14/200000

If I run writes on the Dev cluster, everything runs smoothly - no errors, load is max 1.5-2.
If I reboot the 2 production instances, I can run reads and writes for a while before the crash occurs, otherwise given that machines have been up for 2+ days they crash occurs within minutes.

Any suggestions, ideas would be highly appreciated.

Thanks

Posted 1 year ago #
jbellis
Moderator

Can you check which threads are consuming the CPU, as outlined here? http://spyced.blogspot.com/2010/01/linux-performance-basics.html

(Also worth checking the other performance metrics described on that post.)

Posted 1 year ago #
Anonymous
Unregistered

Already checked the metrics. As I mentioned earlier, disk utilization is very low, especially for writes ( writes is about 0.0* MB/sec). Heap size is 4GB and I'm not getting any OOM exceptions. "Top" doesn't provide much useful information in this case.

This is an output of jstack

"Thrift:21" daemon prio=10 tid=0x0000000040aae000 nid=0x1580 runnable [0x00007f6fdd2d6000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x0000000712909f50> (a java.io.BufferedInputStream)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

"Thrift:20" daemon prio=10 tid=0x0000000040859000 nid=0x157f runnable [0x00007f8548107000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x000000070626b468> (a java.io.BufferedInputStream)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

This is the dominant thread and we have bunch of these in the output. It seems to me that a possible bottle neck might be in the Thrift API, reading all the write request ...

And, as I mentioned system.log is clean. Also, the overload occurs so quickly that there's not much time I can spend for investigation. Once it starts increasing, the machine crashes withing seconds.

Posted 1 year ago #
jbellis
Moderator

If the machine itself is crashing you have an OS- or hardware- level problem, perhaps this one: http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs

Posted 1 year ago #

RSS feed for this topic