OutOfMemory problem

(14 posts) (2 voices)

• Started 1 year ago by
• Latest reply from joaquin
• This topic is not resolved

Tags:

• prom dresses in london

1. 

   Anonymous
   Unregistered

   Hi,

   I am using Cassandra 1.0.8 which should already have fix for CASSANDRA-3492.
   I have counters column family which is about 4GB and it has enabled compression with following settings: compression_options = {sstable_compression:SnappyCompressor, chunk_length_kb:64}. I am running it on my laptop with 4GB of RAM, with all default settings, and looks like Cassandra gets 1GB of RAM.

   I have program which continuously inserts about 1k rows per second and it works Ok. But when I am trying to do simple queries like "list countersCF limit 10" or "get countersCF[utf8('hello')]" in CLI, it gives me TimeOut exception and internal errors.
   I looked at cassandra logs, and see following:

   java.lang.OutOfMemoryError: Java heap space
   at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:64)
   at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:44)
   at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:949)
   at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:72)
   at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:796)
   at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:88)
   at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1329)
   at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:806)
   at org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:697)
   at org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.process(Cassandra.java:3202)
   at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
   at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
   at java.lang.Thread.run(Thread.java:636)

   Does any body have any idea what's wrong? How can I better monitor, diagnose and prevent this problem?

   Thanks.

   Posted 1 year ago #
2. 

   joaquin
   Administrator

   Hello,

   When experiencing OOMs, your java heap space is not tuned efficiently. Try following this guid to increase your heap space:
   http://www.datastax.com/docs/1.0/operations/tuning#tuning-java-heap-size

   With a heap size of 1, you're already 0.3 GB over according to the above formula (with memtables and the 1 GB requirement alone).

   Running on a mac should only be considered for running for development. Perhaps starting a Rackspace or EC2 instance with 4GB with only Cassandra running may allow you to stress the machine a bit more.

   Thanks,
   Joaquin

   Posted 1 year ago #
3. 

   Anonymous
   Unregistered

   Thank you for quick response!
   Actually I already tried the same on 4G RAM VM and with the same result. But after I disabled compression and recreated my data, I started getting responses for my queries, so the problem maybe in compression.

   Bad thing there is that latency of those queries is horrible - about 20secs. It looks unreasonable. I increased HEAP MAX to 2GB, and I don't see any complains about HEAP in the
   logs. So why my queries are so slow? How can I trace what's going on. Any ideas?

   Thanks again.

   Posted 1 year ago #
4. 

   joaquin
   Administrator

   Could you explain what your read and write workloads look like?

   Thanks,
   Joaquin

   Posted 1 year ago #
5. 

   joaquin
   Administrator

   Actually, I realized that you already had that information on the first response. Are there any other workloads running besides whats mentioned above?

   Compression shouldn't cause this much latency.

   Perhaps rpc_timeout in cassandra.yaml has been increased?

   What other changes to the cassandra.yaml have been performed?

   Thanks,
   Joaquin

   Posted 1 year ago #
6. 

   Anonymous
   Unregistered

   I didn't do any changes in cassandra.yaml at all.
   I suspect what happened is that my inserts throughput was very high, and DB didn't have enough time for compaction and created huge number of sstables, which makes lookup time big. But 20 secs(and even more in most of the times) for response still seems too high for me.
   Any ideas for monitoring this would be very helpful.

   Posted 1 year ago #
7. 

   joaquin
   Administrator

   20 second look ups is definitely way to high. Could you confirm your rpc_timeout is still set to 10?

   Also, perhaps there are ERRORs or WARNs in your /var/log/cassandra/system.log that may give some extra information. Could you check this?

   Thanks,
   Joaquin

   Posted 1 year ago #
8. 

   Anonymous
   Unregistered

   Here is my cassandra.yml timeout setting:

   rpc_timeout_in_ms: 10000

   When I said that latency is about 20s, I meant that I am doing few retries, because first time it's dying with TimeOut exception.

   Indeed I see some warnings in logs like:

   WARN [ScheduledTasks:1] 2012-04-10 14:19:21,490 GCInspector.java (line 145) Heap is 0.8483300063982094 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't w
   ant Cassandra to do this automatically
   WARN [ScheduledTasks:1] 2012-04-10 14:19:21,491 StorageService.java (line 2651) Flushing CFS(Keyspace='processed', ColumnFamily='stepsCF') to relieve memory pressure

   Posted 1 year ago #
9. 

   joaquin
   Administrator

   That should be the real culprit:

   Here's the guide to tuning your Java heap size effectively:
   http://www.datastax.com/docs/1.0/operations/tuning#tuning-java-heap-size

   After tuning this appropriately, let us know if you still see these issues.

   Thanks,
   Joaquin

   Posted 1 year ago #
10. 

    Anonymous
    Unregistered

    Ok, my current memory settings are following:

    HEAP size: 2G
    memtable_total_space_in_mb - default = 2G / 3 = 650MB
    estimated key cache size: (1662 + 64) * 1662 * 10 - 12 = 28MB

    So looks like my current config feets formula from your link.

    But I am still getting 40s latency on cold queries.
    I have ongoing compaction process, maybe it will solve the problems. But in a meanwhile maybe there is something I can look at(useful jmx counters, nodetool output, etc)?

    Posted 1 year ago #
11. 

    Anonymous
    Unregistered

    Looks like the problem was the pure compaction.
    Probably I sent too much inserts and my instance couldn't handle throughput and compact efficiently.
    It works much faster(average latency is 3ms) after major compaction.

    Posted 1 year ago #
12. 

    joaquin
    Administrator

    Major compaction should never be performed in production, except for some very rare circumstances. The reason being that once a major compaction is performed, automatic minor compactions will never be optimized for what they need to do since one of the tables is now incredibly large. In production, once a major compaction is performed, weekly major compactions have to be scheduled, and only during downtimes as major compactions are CPU and IO intensive.

    According to your heap space, it would have come out to use an estimated 1.68 GB of your 2GB. This percentage of occupied memory comes out to 84%, which is the same as the warning above. The memory pressure warnings that you are getting are causing flushes of your memtables out to sstables at a higher rate thereby creating multiple sstables and thus triggering compaction.

    Here's a good guide to compaction tuning in general:
    http://www.datastax.com/docs/1.0/operations/tuning#tuning-options-for-size-tiered-compaction

    But really, you want to start with the bud of the problem which states that Cassandra doesn't have a enough heap to operate.

    Currently the Cassandra distribution for Mac is for developer only use meaning that you shouldn't see these same issues when run a machine of 8-16GB. From what I've seen, in order for Cassandra to run with a minimal amount of smoothness for basic stress testing, a minimum of 8GB gets you more realistic operations.

    Thanks,
    Joaquin

    Posted 1 year ago #
13. 

    Anonymous
    Unregistered

    I started major compaction because after server restart(when I added more memory) cassandra stopped doing automatic compaction, and I didn't know how to force it to start automatic compaction again.
    Looks like it can trigger minor compaction only when some sstable is flushed to disk, but in my case I stopped write anything to cassandra. If you know another way how to trigger automatic compaction, it would be very helpful.

    Posted 1 year ago #
14. 

    joaquin
    Administrator

    Hello,

    Aside from tuning your compaction settings, there are no other ways to manually trigger automatic compaction. If the sstables were within the min and max compaction thresholds compaction would have occurred if waited long enough.

    Thanks,
    Joaquin

    Posted 1 year ago #

RSS feed for this topic