DataStax Support Forums » Topic: Cassandra - java.lang.OutOfMemoryError: Java heap space

DataStax Support Forums » Topic: Cassandra - java.lang.OutOfMemoryError: Java heap space http://www.datastax.com/support-forums/topic/cassandra-javalangoutofmemoryerror-java-heap-space Software, Support, and Training for Apache Cassandra en-US Sat, 18 May 2013 14:13:53 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php joaquin on "Cassandra - java.lang.OutOfMemoryError: Java heap space" http://www.datastax.com/support-forums/topic/cassandra-javalangoutofmemoryerror-java-heap-space#post-1443 Wed, 28 Mar 2012 17:05:30 +0000 joaquin 1443@http://www.datastax.com/support-forums/ Hello, 1. Even though this article is for 0.7.x, it should help answer more thoroughly how memtable_throughput_in_mb and internal caches should be calculated: <a href="http://www.datastax.com/docs/0.7/operations/tuning#heap-sizing" rel="nofollow">http://www.datastax.com/docs/0.7/operations/tuning#heap-sizing</a> 2. You can try by increasing the key cache capacity by 2x and see where that puts you. If you do this, make sure to do this in conjunction with step 1 to ensure your tune with this new key cache capacity. Also, try not to set your heap size to anything larger than 8GB. How much RAM is on your machine? Thanks, Joaquin Anonymous on "Cassandra - java.lang.OutOfMemoryError: Java heap space" http://www.datastax.com/support-forums/topic/cassandra-javalangoutofmemoryerror-java-heap-space#post-1424 Thu, 22 Mar 2012 16:25:26 +0000 Anonymous 1424@http://www.datastax.com/support-forums/ Hi Jbellis, Thanks for your quick response. I'm trying to avoid cassandra OOM exception. heap size = memtable_throughput_in_mb * 3 * number of hot CFs + 1G + internal caches 1. Based on your formula, we can calculate the heap size. But Where can i get memtable_throughput_in_mb and internal caches To calculate the heap size? 2.Below Key cache hit rate value is high. To improve the performance, How much i need to set by using setcachecapacity? My cf statistics: Keyspace: beacon Read Count: 58098 Read Latency: 3.6910174704809116 ms. Write Count: 2875 Write Latency: 0.015630956521739132 ms. Pending Tasks: 0 Column Family: Alarms SSTable count: 2 Space used (live): 5133969 Space used (total): 5133969 Memtable Columns Count: 0 Memtable Data Size: 0 Memtable Switch Count: 0 Read Count: 68 Read Latency: 9.724 ms. Write Count: 0 Write Latency: NaN ms. Pending Tasks: 0 Key cache capacity: 3264 Key cache size: 6 Key cache hit rate: 0.9491525423728814 Row cache: disabled Compacted row minimum size: 0 Compacted row maximum size: 0 Compacted row mean size: 0 Kindly provide your valuable feedback asap to improve the performance. Thanks, Ramesh Kasi jbellis on "Cassandra - java.lang.OutOfMemoryError: Java heap space" http://www.datastax.com/support-forums/topic/cassandra-javalangoutofmemoryerror-java-heap-space#post-1387 Sun, 18 Mar 2012 20:21:17 +0000 jbellis 1387@http://www.datastax.com/support-forums/ The latest 0.6 release is 0.6.13: <a href="http://archive.apache.org/dist/cassandra/0.6.13/" rel="nofollow">http://archive.apache.org/dist/cassandra/0.6.13/</a> Please follow the same troubleshooting guide I linked above and post your findings. Anonymous on "Cassandra - java.lang.OutOfMemoryError: Java heap space" http://www.datastax.com/support-forums/topic/cassandra-javalangoutofmemoryerror-java-heap-space#post-1385 Sun, 18 Mar 2012 18:40:03 +0000 Anonymous 1385@http://www.datastax.com/support-forums/ Hello Bellis, Have found some info from the forums, after upgraded to cassandra 0.6.1 still the user facing the same problem when adding more rows/columns. So it means cassandra is not consistent? I think, Increasing the JVM Heap Space and upgrading to latest version will not resolve the above issue. Please give up the solution to resolve the issue quickly... <a href="http://permalink.gmane.org/gmane.comp.db.cassandra.user/5017" rel="nofollow">http://permalink.gmane.org/gmane.comp.db.cassandra.user/5017</a> Thanks in advance... jbellis on "Cassandra - java.lang.OutOfMemoryError: Java heap space" http://www.datastax.com/support-forums/topic/cassandra-javalangoutofmemoryerror-java-heap-space#post-1378 Thu, 15 Mar 2012 16:29:42 +0000 jbellis 1378@http://www.datastax.com/support-forums/ Upgrade to the latest 0.6 release, then follow the advice here: <a href="http://www.datastax.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors" rel="nofollow">http://www.datastax.com/docs/0.6/troubleshooting/index#nodes-are-dying-with-oom-errors</a> Anonymous on "Cassandra - java.lang.OutOfMemoryError: Java heap space" http://www.datastax.com/support-forums/topic/cassandra-javalangoutofmemoryerror-java-heap-space#post-1376 Thu, 15 Mar 2012 11:33:19 +0000 Anonymous 1376@http://www.datastax.com/support-forums/ Hi, We are currently using cassandra ver 0.6.2 in Production. The issue we are having is OOM (java.lang.OutOfMemoryError: Java heap space) Below i have added the logs and cassandra configuration to identify the issue. Log: -------- INFO [COMPACTION-POOL:1] 2012-02-20 22:14:36,715 CompactionManager.java (line 246) Compacting [org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-4833-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-5478-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-6085-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-7065-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/Traps-11706-Data.db')] INFO [GC inspection] 2012-02-20 22:14:51,307 GCInspector.java (line 110) GC for ConcurrentMarkSweep: 1548 ms, 462469616 reclaimed leaving 576773008 used; max is 2277179392 INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,310 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/Alarms-13823-Data.db INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,311 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/CircuitID-756-Data.db INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,311 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/CircuitID-757-Data.db INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,312 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/CircuitID-755-Data.db INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,312 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/Alarms-13824-Data.db INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,312 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/Alarms-13825-Data.db INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,312 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/AlertDate-13806-Data.db INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,313 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/AlertDate-13805-Data.db INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,318 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/Alarms-13822-Data.db INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,321 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/AlertDate-13803-Data.db INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,321 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/CircuitID-758-Data.db INFO [SSTABLE-CLEANUP-TIMER] 2012-02-20 22:15:01,321 SSTableDeletingReference.java (line 104) Deleted /cassandra/data/beacon/AlertDate-13804-Data.db INFO [GC inspection] 2012-02-20 22:16:08,795 GCInspector.java (line 110) GC for ConcurrentMarkSweep: 3650 ms, 125120 reclaimed leaving 1917886152 used; max is 2277179392 ERROR [COMPACTION-POOL:1] 2012-02-20 22:16:08,798 DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86) at org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:887) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2786) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94) at java.io.DataOutputStream.write(DataOutputStream.java:90) at java.io.FilterOutputStream.write(FilterOutputStream.java:80) at org.apache.cassandra.db.ColumnSerializer.writeName(ColumnSerializer.java:39) at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:61) at org.apache.cassandra.db.SuperColumnSerializer.serialize(SuperColumn.java:311) at org.apache.cassandra.db.SuperColumnSerializer.serialize(SuperColumn.java:284) at org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:87) at org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:99) at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:131) at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:41) at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130) at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183) at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94) at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:299) at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102) at org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) ... 2 more INFO [COMPACTION-POOL:1] 2012-02-20 22:16:08,798 CompactionManager.java (line 246) Compacting [org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/AlertGroups-13840-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/AlertGroups-13841-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/AlertGroups-13842-Data.db'),org.apache.cassandra.io.SSTableReader(path='/cassandra/data/beacon/AlertGroups-13843-Data.db')] INFO [COMPACTION-POOL:1] 2012-02-20 22:16:08,913 CompactionManager.java (line 326) Compacted to /cassandra/data/beacon/AlertGroups-13844-Data.db. 1863395/1575561 bytes for 2554 keys. Time: 114ms. INFO [FLUSH-TIMER] 2012-02-20 23:14:33,889 ColumnFamilyStore.java (line 357) ActiveGroups has reached its threshold; switching in a fresh Memtable at CommitLogContext(file='/var/cassandra/commitlog/CommitLog-1329745139372.log', position=2624553) cassandra configuration: ------------------------ # Arguments to pass to the JVM JVM_OPTS=" \ -ea \ -Xms256M \ -Xmx6G \ -XX:+UseParNewGC \ -XX:+UseConcMarkSweepGC \ -XX:+CMSParallelRemarkEnabled \ -XX:SurvivorRatio=8 \ -XX:MaxTenuringThreshold=1 \ -XX:+HeapDumpOnOutOfMemoryError \ -Dcom.sun.management.jmxremote.port=8080 \ -Dcom.sun.management.jmxremote.ssl=false \ -Dcom.sun.management.jmxremote.authenticate=false" Please advice me how to resolve this issue. Thanks Ramesh Kasi