Support Forums : DataStax

DataStax Support Forums » Apache Cassandra

Cassandra node going down

(3 posts) (2 voices)

Started 8 months ago by rohit.reddy@insideview.com
Latest reply from rohit.reddy@insideview.com
This topic is resolved

Tags:

No tags yet.

rohit.reddy@insideview.com
Member

Hi,

I'm facing a problem in Cassandra cluster deployed on EC2 where the node is going down under write load.

I have configured a cluster of 4 Large EC2 nodes with RF of 2.
All nodes are instance storage backed. DISK is RAID0 with 800GB

I'm pumping in write requests at about 4000 writes/sec. One of the node went down under this load. The total data size in each node was not more than 7GB
Got the following WARN messages in the LOG file...

1. setting live ratio to minimum of 1.0 instead of 0.9003153296009601
2. Heap is 0.7515559786053904 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do
this automatically
3. WARN [CompactionExecutor:570] 2012-09-14 11:45:12,024 CompactionTask.java (line 84) insufficient space to compact all requested files

All cassandra settings are default settings.
Do i need to tune anything to support this write rate?

Thanks
Rohit

Posted 8 months ago #
thobbs
Administrator

That last log message indicates that you are basically out of space on whatever filesystem the cassandra data directory is on. Make sure you've mounted the raid properly and that data_file_directories points to the correct location in cassandra.yaml. I would guess that you're accidentally using the same small disk that holds the os for that instance.

Posted 8 months ago #
rohit.reddy@insideview.com
Member

Thanks for the reply. The disk on the ec2 node failed, which led to this.

Posted 8 months ago #

RSS feed for this topic