DataStax Support Forums » Topic: OutOfMemory problem

DataStax Support Forums » Topic: OutOfMemory problem http://www.datastax.com/support-forums/topic/outofmemory-problem Software, Support, and Training for Apache Cassandra en-US Mon, 20 May 2013 02:06:57 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php joaquin on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1642 Mon, 16 Apr 2012 20:57:11 +0000 joaquin 1642@http://www.datastax.com/support-forums/ Hello, Aside from tuning your compaction settings, there are no other ways to manually trigger automatic compaction. If the sstables were within the min and max compaction thresholds compaction would have occurred if waited long enough. Thanks, Joaquin Anonymous on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1641 Mon, 16 Apr 2012 20:52:40 +0000 Anonymous 1641@http://www.datastax.com/support-forums/ I started major compaction because after server restart(when I added more memory) cassandra stopped doing automatic compaction, and I didn't know how to force it to start automatic compaction again. Looks like it can trigger minor compaction only when some sstable is flushed to disk, but in my case I stopped write anything to cassandra. If you know another way how to trigger automatic compaction, it would be very helpful. joaquin on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1633 Mon, 16 Apr 2012 17:38:18 +0000 joaquin 1633@http://www.datastax.com/support-forums/ Major compaction should never be performed in production, except for some very rare circumstances. The reason being that once a major compaction is performed, automatic minor compactions will never be optimized for what they need to do since one of the tables is now incredibly large. In production, once a major compaction is performed, weekly major compactions have to be scheduled, and only during downtimes as major compactions are CPU and IO intensive. According to your heap space, it would have come out to use an estimated 1.68 GB of your 2GB. This percentage of occupied memory comes out to 84%, which is the same as the warning above. The memory pressure warnings that you are getting are causing flushes of your memtables out to sstables at a higher rate thereby creating multiple sstables and thus triggering compaction. Here's a good guide to compaction tuning in general: <a href="http://www.datastax.com/docs/1.0/operations/tuning#tuning-options-for-size-tiered-compaction" rel="nofollow">http://www.datastax.com/docs/1.0/operations/tuning#tuning-options-for-size-tiered-compaction</a> But really, you want to start with the bud of the problem which states that Cassandra doesn't have a enough heap to operate. Currently the Cassandra distribution for Mac is for developer only use meaning that you shouldn't see these same issues when run a machine of 8-16GB. From what I've seen, in order for Cassandra to run with a minimal amount of smoothness for basic stress testing, a minimum of 8GB gets you more realistic operations. Thanks, Joaquin Anonymous on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1632 Mon, 16 Apr 2012 17:08:15 +0000 Anonymous 1632@http://www.datastax.com/support-forums/ Looks like the problem was the pure compaction. Probably I sent too much inserts and my instance couldn't handle throughput and compact efficiently. It works much faster(average latency is 3ms) after major compaction. Anonymous on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1616 Fri, 13 Apr 2012 13:06:05 +0000 Anonymous 1616@http://www.datastax.com/support-forums/ Ok, my current memory settings are following: HEAP size: 2G memtable_total_space_in_mb - default = 2G / 3 = 650MB estimated key cache size: (1662 + 64) * 1662 * 10 - 12 = 28MB So looks like my current config feets formula from your link. But I am still getting 40s latency on cold queries. I have ongoing compaction process, maybe it will solve the problems. But in a meanwhile maybe there is something I can look at(useful jmx counters, nodetool output, etc)? joaquin on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1581 Tue, 10 Apr 2012 23:02:18 +0000 joaquin 1581@http://www.datastax.com/support-forums/ That should be the real culprit: Here's the guide to tuning your Java heap size effectively: <a href="http://www.datastax.com/docs/1.0/operations/tuning#tuning-java-heap-size" rel="nofollow">http://www.datastax.com/docs/1.0/operations/tuning#tuning-java-heap-size</a> After tuning this appropriately, let us know if you still see these issues. Thanks, Joaquin Anonymous on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1578 Tue, 10 Apr 2012 21:23:18 +0000 Anonymous 1578@http://www.datastax.com/support-forums/ Here is my cassandra.yml timeout setting: rpc_timeout_in_ms: 10000 When I said that latency is about 20s, I meant that I am doing few retries, because first time it's dying with TimeOut exception. Indeed I see some warnings in logs like: WARN [ScheduledTasks:1] 2012-04-10 14:19:21,490 GCInspector.java (line 145) Heap is 0.8483300063982094 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't w ant Cassandra to do this automatically WARN [ScheduledTasks:1] 2012-04-10 14:19:21,491 StorageService.java (line 2651) Flushing CFS(Keyspace='processed', ColumnFamily='stepsCF') to relieve memory pressure joaquin on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1565 Tue, 10 Apr 2012 17:51:48 +0000 joaquin 1565@http://www.datastax.com/support-forums/ 20 second look ups is definitely way to high. Could you confirm your rpc_timeout is still set to 10? Also, perhaps there are ERRORs or WARNs in your /var/log/cassandra/system.log that may give some extra information. Could you check this? Thanks, Joaquin Anonymous on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1558 Tue, 10 Apr 2012 04:49:36 +0000 Anonymous 1558@http://www.datastax.com/support-forums/ I didn't do any changes in cassandra.yaml at all. I suspect what happened is that my inserts throughput was very high, and DB didn't have enough time for compaction and created huge number of sstables, which makes lookup time big. But 20 secs(and even more in most of the times) for response still seems too high for me. Any ideas for monitoring this would be very helpful. joaquin on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1557 Mon, 09 Apr 2012 23:06:28 +0000 joaquin 1557@http://www.datastax.com/support-forums/ Actually, I realized that you already had that information on the first response. Are there any other workloads running besides whats mentioned above? Compression shouldn't cause this much latency. Perhaps rpc_timeout in cassandra.yaml has been increased? What other changes to the cassandra.yaml have been performed? Thanks, Joaquin joaquin on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1556 Mon, 09 Apr 2012 22:59:05 +0000 joaquin 1556@http://www.datastax.com/support-forums/ Could you explain what your read and write workloads look like? Thanks, Joaquin Anonymous on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1552 Mon, 09 Apr 2012 03:56:44 +0000 Anonymous 1552@http://www.datastax.com/support-forums/ Thank you for quick response! Actually I already tried the same on 4G RAM VM and with the same result. But after I disabled compression and recreated my data, I started getting responses for my queries, so the problem maybe in compression. Bad thing there is that latency of those queries is horrible - about 20secs. It looks unreasonable. I increased HEAP MAX to 2GB, and I don't see any complains about HEAP in the logs. So why my queries are so slow? How can I trace what's going on. Any ideas? Thanks again. joaquin on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1551 Sun, 08 Apr 2012 21:17:04 +0000 joaquin 1551@http://www.datastax.com/support-forums/ Hello, When experiencing OOMs, your java heap space is not tuned efficiently. Try following this guid to increase your heap space: <a href="http://www.datastax.com/docs/1.0/operations/tuning#tuning-java-heap-size" rel="nofollow">http://www.datastax.com/docs/1.0/operations/tuning#tuning-java-heap-size</a> With a heap size of 1, you're already 0.3 GB over according to the above formula (with memtables and the 1 GB requirement alone). Running on a mac should only be considered for running for development. Perhaps starting a Rackspace or EC2 instance with 4GB with only Cassandra running may allow you to stress the machine a bit more. Thanks, Joaquin Anonymous on "OutOfMemory problem" http://www.datastax.com/support-forums/topic/outofmemory-problem#post-1550 Sun, 08 Apr 2012 17:19:54 +0000 Anonymous 1550@http://www.datastax.com/support-forums/ Hi, I am using Cassandra 1.0.8 which should already have fix for CASSANDRA-3492. I have counters column family which is about 4GB and it has enabled compression with following settings: compression_options = {sstable_compression:SnappyCompressor, chunk_length_kb:64}. I am running it on my laptop with 4GB of RAM, with all default settings, and looks like Cassandra gets 1GB of RAM. I have program which continuously inserts about 1k rows per second and it works Ok. But when I am trying to do simple queries like "list countersCF limit 10" or "get countersCF[utf8('hello')]" in CLI, it gives me TimeOut exception and internal errors. I looked at cassandra logs, and see following: java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:64) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:44) at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:949) at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:72) at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:796) at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:88) at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1329) at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:806) at org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:697) at org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.process(Cassandra.java:3202) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Does any body have any idea what's wrong? How can I better monitor, diagnose and prevent this problem? Thanks.