Support Forums : DataStax

DataStax Support Forums » Apache Cassandra

Cassandra - java.lang.OutOfMemoryError: Java heap space

(6 posts) (4 voices)

Started 1 year ago by
Latest reply from joaquin
This topic is not a support question

Tags:

No tags yet.

Anonymous
Unregistered

Hi,

We are currently using cassandra ver 0.6.2 in Production.
The issue we are having is OOM (java.lang.OutOfMemoryError: Java heap space)
Below i have added the logs and cassandra configuration to identify the issue.

Log:
--------

INFO [COMPACTION-POOL:1] 2012-02-20 22:14:36,715 CompactionManager.java (line 246) Compacting [org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-4833-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-5478-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-6085-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-7065-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-11706-Data.db')]
INFO [GC inspection] 2012-02-20 22:14:51,307 GCInspector.java (line 110) GC for ConcurrentMarkSweep: 1548 ms, 462469616 reclaimed leaving 576773008 used; max is 2277179392
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,310 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/Alarms-13823-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,311 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/CircuitID-756-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,311 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/CircuitID-757-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,312 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/CircuitID-755-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,312 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/Alarms-13824-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,312 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/Alarms-13825-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,312 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/AlertDate-13806-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,313 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/AlertDate-13805-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,318 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/Alarms-13822-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,321 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/AlertDate-13803-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,321 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/CircuitID-758-Data.db
INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,321 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/AlertDate-13804-Data.db
INFO [GC inspection] 2012-02-20 22:16:08,795 GCInspector.java (line 110) GC for ConcurrentMarkSweep: 3650 ms, 125120 reclaimed leaving 1917886152 used; max is 2277179392
ERROR [COMPACTION-POOL:1] 2012-02-20 22:16:08,798 DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:887)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
at org.apache.cassandra.db.ColumnSerializer.writeName(ColumnSerializer.java:39)
at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:61)
at org.apache.cassandra.db.SuperColumnSerializer.serialize(SuperColumn.java:311)
at org.apache.cassandra.db.SuperColumnSerializer.serialize(SuperColumn.java:284)
at org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:87)
at org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:99)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:131)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:41)
at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:299)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
... 2 more
INFO [COMPACTION-POOL:1] 2012-02-20 22:16:08,798 CompactionManager.java (line 246) Compacting [org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/AlertGroups-13840-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/AlertGroups-13841-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/AlertGroups-13842-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/AlertGroups-13843-Data.db')]
INFO [COMPACTION-POOL:1] 2012-02-20 22:16:08,913 CompactionManager.java (line 326) Compacted to /cassandra/data/beacon/AlertGroups-13844-Data.db. 1863395/1575561 bytes for 2554 keys. Time: 114ms.
INFO [FLUSH-TIMER] 2012-02-20 23:14:33,889 ColumnFamilyStore.java (line 357) ActiveGroups has reached its threshold; switching in a fresh Memtable at CommitLogContext(file='/var/cassandra/commitlog/CommitLog-1329745139372.log', position=2624553)

cassandra configuration:
------------------------

# Arguments to pass to the JVM
JVM_OPTS=" \
-ea \
-Xms256M \
-Xmx6G \
-XX:+UseParNewGC \
-XX:+UseConcMarkSweepGC \
-XX:+CMSParallelRemarkEnabled \
-XX:SurvivorRatio=8 \
-XX:MaxTenuringThreshold=1 \
-XX:+HeapDumpOnOutOfMemoryError \
-Dcom.sun.management.jmxremote.port=8080 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false"

Please advice me how to resolve this issue.

Thanks
Ramesh Kasi

Posted 1 year ago #
jbellis
Moderator

Upgrade to the latest 0.6 release, then follow the advice here: http://www.datastax.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors

Posted 1 year ago #
Anonymous
Unregistered

Hello Bellis,

Have found some info from the forums, after upgraded to cassandra 0.6.1 still the user facing the same problem when adding more rows/columns. So it means cassandra is not consistent? I think, Increasing the JVM Heap Space and upgrading to latest version will not resolve the above issue. Please give up the solution to resolve the issue quickly...

http://permalink.gmane.org/gmane.comp.db.cassandra.user/5017

Thanks in advance...

Posted 1 year ago #
jbellis
Moderator

The latest 0.6 release is 0.6.13: http://archive.apache.org/dist/cassandra/0.6.13/

Please follow the same troubleshooting guide I linked above and post your findings.

Posted 1 year ago #
Anonymous
Unregistered

Hi Jbellis,

Thanks for your quick response.

I'm trying to avoid cassandra OOM exception.

heap size = memtable_throughput_in_mb * 3 * number of hot CFs + 1G + internal caches

1. Based on your formula, we can calculate the heap size.
But Where can i get memtable_throughput_in_mb and internal caches To calculate the heap size?

2.Below Key cache hit rate value is high. To improve the performance, How much i need to set by using setcachecapacity?

My cf statistics:
Keyspace: beacon
Read Count: 58098
Read Latency: 3.6910174704809116 ms.
Write Count: 2875
Write Latency: 0.015630956521739132 ms.
Pending Tasks: 0
Column Family: Alarms
SSTable count: 2
Space used (live): 5133969
Space used (total): 5133969
Memtable Columns Count: 0
Memtable Data Size: 0
Memtable Switch Count: 0
Read Count: 68
Read Latency: 9.724 ms.
Write Count: 0
Write Latency: NaN ms.
Pending Tasks: 0
Key cache capacity: 3264
Key cache size: 6
Key cache hit rate: 0.9491525423728814
Row cache: disabled
Compacted row minimum size: 0
Compacted row maximum size: 0
Compacted row mean size: 0

Kindly provide your valuable feedback asap to improve the performance.

Thanks,
Ramesh Kasi

Posted 1 year ago #
joaquin
Administrator

Hello,

1. Even though this article is for 0.7.x, it should help answer more thoroughly how memtable_throughput_in_mb and internal caches should be calculated:
http://www.datastax.com/docs/0.7/operations/tuning#heap-sizing

2. You can try by increasing the key cache capacity by 2x and see where that puts you. If you do this, make sure to do this in conjunction with step 1 to ensure your tune with this new key cache capacity.

Also, try not to set your heap size to anything larger than 8GB. How much RAM is on your machine?

Thanks,
Joaquin

Posted 1 year ago #

RSS feed for this topic