Hi,
We are currently using cassandra ver 0.6.2 in Production.
The issue we are having is OOM (java.lang.OutOfMemoryError: Java heap space)
Below i have added the logs and cassandra configuration to identify the issue.
Log:
--------
INFO [COMPACTION-POOL:1] 2012-02-20 22:14:36,715 CompactionManager.java (line 246) Compacting [org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-4833-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-5478-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-6085-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-7065-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-11706-Data.db')]
INFO [GC inspection] 2012-02-20 22:14:51,307 GCInspector.java (line 110) GC for ConcurrentMarkSweep: 1548 ms, 462469616 reclaimed leaving 576773008 used; max is 2277179392
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,310 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/Alarms-13823-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,311 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/CircuitID-756-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,311 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/CircuitID-757-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,312 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/CircuitID-755-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,312 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/Alarms-13824-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,312 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/Alarms-13825-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,312 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/AlertDate-13806-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,313 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/AlertDate-13805-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,318 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/Alarms-13822-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,321 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/AlertDate-13803-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,321 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/CircuitID-758-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,321 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/AlertDate-13804-Data.db
INFO [GC inspection] 2012-02-20 22:16:08,795 GCInspector.java (line 110) GC for ConcurrentMarkSweep: 3650 ms, 125120 reclaimed leaving 1917886152 used; max is 2277179392
ERROR [COMPACTION-POOL:1] 2012-02-20 22:16:08,798 DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:887)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
at org.apache.cassandra.db.ColumnSerializer.writeName(ColumnSerializer.java:39)
at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:61)
at org.apache.cassandra.db.SuperColumnSerializer.serialize(SuperColumn.java:311)
at org.apache.cassandra.db.SuperColumnSerializer.serialize(SuperColumn.java:284)
at org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:87)
at org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:99)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:131)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:41)
at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:299)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
... 2 more
INFO [COMPACTION-POOL:1] 2012-02-20 22:16:08,798 CompactionManager.java (line 246) Compacting [org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/AlertGroups-13840-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/AlertGroups-13841-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/AlertGroups-13842-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/AlertGroups-13843-Data.db')]
INFO [COMPACTION-POOL:1] 2012-02-20 22:16:08,913 CompactionManager.java (line 326) Compacted to /cassandra/data/beacon/AlertGroups-13844-Data.db. 1863395/1575561 bytes for 2554 keys. Time: 114ms.
INFO [FLUSH-TIMER] 2012-02-20 23:14:33,889 ColumnFamilyStore.java (line 357) ActiveGroups has reached its threshold; switching in a fresh Memtable at CommitLogContext(file='/var/cassandra/commitlog/CommitLog-1329745139372.log', position=2624553)
cassandra configuration:
------------------------
# Arguments to pass to the JVM
JVM_OPTS=" \
-ea \
-Xms256M \
-Xmx6G \
-XX:+UseParNewGC \
-XX:+UseConcMarkSweepGC \
-XX:+CMSParallelRemarkEnabled \
-XX:SurvivorRatio=8 \
-XX:MaxTenuringThreshold=1 \
-XX:+HeapDumpOnOutOfMemoryError \
-Dcom.sun.management.jmxremote.port=8080 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false"
Please advice me how to resolve this issue.
Thanks
Ramesh Kasi
