DataStax Support Forums » Topic: slow compaction

DataStax Support Forums » Topic: slow compaction http://www.datastax.com/support-forums/topic/slow-compaction Software, Support, and Training for Apache Cassandra en-US Fri, 24 May 2013 08:23:42 +0000 http://bbpress.org/?v=1.0.3 <![CDATA[Search]]> q http://www.datastax.com/support-forums/search.php Anonymous on "slow compaction" http://www.datastax.com/support-forums/topic/slow-compaction#post-1336 Thu, 08 Mar 2012 06:00:29 +0000 Anonymous 1336@http://www.datastax.com/support-forums/ Hi Joaquin, I started a major compaction manually and it only finished 12% in 2 days. Is there a way to speed this up? I tried to reduce compaction_throughput_mb_per_sec, but it keeps crashing cassandra. pending tasks: 1119 compaction type keyspace column family bytes compacted bytes total progress Major radarks RadarEvents 88834651255 757236269930 11.73% joaquin on "slow compaction" http://www.datastax.com/support-forums/topic/slow-compaction#post-1329 Wed, 07 Mar 2012 19:41:14 +0000 joaquin 1329@http://www.datastax.com/support-forums/ Hello Feng, 1. 8GB should be sufficient and is what DataStax recommends. The highest we ever recommend is 12GB for rare circumstances. If there is memory pressure, you want to look at: <a href="http://www.datastax.com/docs/0.8/operations/tuning#tuning-java-heap-size" rel="nofollow">http://www.datastax.com/docs/0.8/operations/tuning#tuning-java-heap-size</a> and lower what is in your heap. 2. For a complete breakdown on Leveled Compaction, check out Jonathan's blog here: <a href="http://www.datastax.com/dev/blog/leveled-compaction-in-apache-cassandra" rel="nofollow">http://www.datastax.com/dev/blog/leveled-compaction-in-apache-cassandra</a> and this link for more in depth compaction tuning settings: <a href="http://www.datastax.com/docs/0.8/operations/tuning#tuning-compaction" rel="nofollow">http://www.datastax.com/docs/0.8/operations/tuning#tuning-compaction</a> Thanks, Joaquin Anonymous on "slow compaction" http://www.datastax.com/support-forums/topic/slow-compaction#post-1306 Tue, 06 Mar 2012 17:42:52 +0000 Anonymous 1306@http://www.datastax.com/support-forums/ regarding 2), the largest SSTable we have is 150GB and top 4 is 480GB. How could Cassandra compact a 580GB data size? -rw-r--r-- 1 cassandra cassandra 49021373086 Feb 20 21:11 /data/cassandra/data/radarks/RadarEvents-g-47568-Data.db -rw-r--r-- 1 cassandra cassandra 50960935975 Feb 21 00:36 /data/cassandra/data/radarks/RadarEvents-g-48402-Data.db -rw-r--r-- 1 cassandra cassandra 53793567207 Feb 21 23:06 /data/cassandra/data/radarks/RadarEvents-g-52984-Data.db -rw-r--r-- 1 cassandra cassandra 55739476304 Feb 23 21:37 /data/cassandra/data/radarks/RadarEvents-g-56456-Data.db -rw-r--r-- 1 cassandra cassandra 63872930369 Feb 20 17:41 /data/cassandra/data/radarks/RadarEvents-g-47485-Data.db -rw-r--r-- 1 cassandra cassandra 67808363541 Feb 15 10:18 /data/cassandra/data/radarks/RadarEvents-g-38921-Data.db -rw-r--r-- 1 cassandra cassandra 68380523771 Mar 2 01:29 /data/cassandra/data/radarks/RadarEvents-g-103981-Data.db -rw-r--r-- 1 cassandra cassandra 70114228783 Feb 21 10:28 /data/cassandra/data/radarks/RadarEvents-g-49872-Data.db -rw-r--r-- 1 cassandra cassandra 99802384643 Feb 16 09:28 /data/cassandra/data/radarks/RadarEvents-g-38913-Data.db -rw-r--r-- 1 cassandra cassandra 149112565032 Feb 16 11:20 /data/cassandra/data/radarks/RadarEvents-g-38923-Data.db Anonymous on "slow compaction" http://www.datastax.com/support-forums/topic/slow-compaction#post-1305 Tue, 06 Mar 2012 17:20:54 +0000 Anonymous 1305@http://www.datastax.com/support-forums/ Hi jbellis, thanks for information. regarding to 1), should increasing java heap size help? Currently I am using 8GB heap on a server with 96GB ram. jbellis on "slow compaction" http://www.datastax.com/support-forums/topic/slow-compaction#post-1300 Tue, 06 Mar 2012 03:20:52 +0000 jbellis 1300@http://www.datastax.com/support-forums/ 1) You're running your heap too full and compaction is pushing it over the edge. Reduce your memory usage first. 2) Size-tiered compaction will attempt to compact that 500GB table whenever there are at least 3 other sstables within 50% of its size. 3) Data that no longer belongs on the nodes can contribute to data sizes being inconsistent across the ring; so can compaction variation. You can run cleanup to deal with the former; the latter is normal unless you run major compaction frequently which we do not recommend. Anonymous on "slow compaction" http://www.datastax.com/support-forums/topic/slow-compaction#post-1294 Mon, 05 Mar 2012 22:18:48 +0000 Anonymous 1294@http://www.datastax.com/support-forums/ Last major compaction on RadarEvents on this hot was done on 12/4/2011 while other 5 nodes were done sometime in Feb 2012. Anonymous on "slow compaction" http://www.datastax.com/support-forums/topic/slow-compaction#post-1293 Mon, 05 Mar 2012 22:17:08 +0000 Anonymous 1293@http://www.datastax.com/support-forums/ here is compactionstats output. pending tasks: 11 compaction type keyspace column family bytes compacted bytes total progress Minor radarks RadarEvents 5717430 8398454 68.08% Minor radarksRadarEventsByUser 569197622 752616934 75.63% Minor radarks RadarEvents 350698175302 571225618773 61.39% Minor radarks RadarEvents 1625498635 14706273975 11.05% Validation radarksRadarEventsByType 3742386130 22168328852 16.88% Minor radarks RadarEvents 79970157 250870841 31.88% Anonymous on "slow compaction" http://www.datastax.com/support-forums/topic/slow-compaction#post-1290 Mon, 05 Mar 2012 19:02:09 +0000 Anonymous 1290@http://www.datastax.com/support-forums/ We have a 6-node ring running 0.8.6. The first node has been crashed about once a week recently. We notice that it keeps running large minor compaction with 500GB data and last major compaction was done 3 months ago. In addition, data is not evenly distributed across nodes even token are evenly split across 6 nodes. Address DC Rack Status State Load Owns Token 141784319550391026443072753096570088106 10.89.74.60 slc rack0 Up Normal 1010.96 GB 16.67% 0 10.2.128.55 phx rack0 Up Normal 458.06 GB 16.67% 28356863910078205288614550619314017621 10.89.74.67 slc rack0 Up Normal 328.41 GB 16.67% 56713727820156410577229101238628035242 10.2.128.56 phx rack0 Up Normal 334.91 GB 16.67% 85070591730234615865843651857942052864 10.89.74.62 slc rack0 Up Normal 327.01 GB 16.67% 113427455640312821154458202477256070485 10.2.128.57 phx rack0 Up Normal 666.64 GB 16.67% 141784319550391026443072753096570088106 INFO [FlushWriter:2] 2012-03-03 15:40:59,429 Memtable.java (line 237) Writing Memtable-pdps@321720209(3131/161417 serialized/live bytes, 101 ops) INFO [ScheduledTasks:1] 2012-03-03 15:40:59,436 MessagingService.java (line 612) 724 READ messages dropped in server lifetime INFO [ScheduledTasks:1] 2012-03-03 15:41:09,404 StatusLogger.java (line 50) Pool Name Active Pending Blocked INFO [ScheduledTasks:1] 2012-03-03 15:41:09,433 StatusLogger.java (line 65) ReadStage 32 133 0 INFO [ScheduledTasks:1] 2012-03-03 15:41:09,441 StatusLogger.java (line 65) RequestResponseStage 0 0 0 INFO [CompactionExecutor:37331] 2012-03-03 15:41:09,446 CompactionManager.java (line 608) Compacted to /data/cassandra/data/OpsCenter/rollups60-tmp-g-17507-Data.db. 54,041 to 43,793 (~81% of original) bytes for 317 keys. Time: 72,781ms. INFO [ScheduledTasks:1] 2012-03-03 15:41:09,447 StatusLogger.java (line 65) ReadRepairStage 0 0 0 INFO [FlushWriter:2] 2012-03-03 15:41:09,467 Memtable.java (line 254) Completed flushing /data/cassandra/data/OpsCenter/pdps-g-121861-Data.db (12309 bytes) INFO [ScheduledTasks:1] 2012-03-03 15:41:09,470 StatusLogger.java (line 65) MutationStage 32 99 0 INFO [FlushWriter:2] 2012-03-03 15:41:09,471 Memtable.java (line 237) Writing Memtable-RadarEventsByUser@1835153567(121918/4885533 serialized/live bytes, 2594 ops) INFO [ScheduledTasks:1] 2012-03-03 15:41:09,481 StatusLogger.java (line 65) ReplicateOnWriteStage 0 0 0 INFO [CompactionExecutor:37331] 2012-03-03 15:41:09,490 CompactionManager.java (line 542) Compacting Minor: [SSTableReader(path='/data/cassandra/data/radarks/RadarEvents-g-117144-Data.db'), SSTableReader(path='/data/cassandra/data/radarks/RadarEvents-g-117105-Data.db'), SSTableReader(path='/data/cassandra/data/radarks/RadarEvents-g-117143-Data.db'), SSTableReader(path='/data/cassandra/data/radarks/RadarEvents-g-117148-Data.db'), SSTableReader(path='/data/cassandra/data/radarks/RadarEvents-g-117149-Data.db'), SSTableReader(path='/data/cassandra/data/radarks/RadarEvents-g-117145-Data.db'), SSTableReader(path='/data/cassandra/data/radarks/RadarEvents-g-117146-Data.db'), SSTableReader(path='/data/cassandra/data/radarks/RadarEvents-g-117114-Data.db'), SSTableReader(path='/data/cassandra/data/radarks/RadarEvents-g-117147-Data.db')] INFO [ScheduledTasks:1] 2012-03-03 15:41:09,514 StatusLogger.java (line 65) GossipStage 1 451 0 INFO [ScheduledTasks:1] 2012-03-03 15:41:09,534 StatusLogger.java (line 65) AntiEntropyStage 0 0 0 INFO [ScheduledTasks:1] 2012-03-03 15:41:17,269 StatusLogger.java (line 65) MigrationStage 0 0 0 INFO [ScheduledTasks:1] 2012-03-03 15:41:17,479 StatusLogger.java (line 65) StreamStage 0 0 0 INFO [ScheduledTasks:1] 2012-03-03 15:41:17,491 StatusLogger.java (line 65) MemtablePostFlusher 1 4 0 INFO [ScheduledTasks:1] 2012-03-03 15:41:17,504 StatusLogger.java (line 65) FILEUTILS-DELETE-POOL 0 0 0 INFO [ScheduledTasks:1] 2012-03-03 15:41:17,504 StatusLogger.java (line 65) FlushWriter 1 4 0 INFO [ScheduledTasks:1] 2012-03-03 15:41:17,507 StatusLogger.java (line 65) MiscStage 0 0 0 INFO [ScheduledTasks:1] 2012-03-03 15:41:17,507 StatusLogger.java (line 65) FlushSorter 0 0 0 INFO [ScheduledTasks:1] 2012-03-03 15:41:17,517 StatusLogger.java (line 65) InternalResponseStage 0 0 0 INFO [ScheduledTasks:1] 2012-03-03 15:41:17,528 StatusLogger.java (line 65) HintedHandoff 0 0 0 INFO [ScheduledTasks:1] 2012-03-03 15:41:17,549 StatusLogger.java (line 69) CompactionManager n/a 73 INFO [ScheduledTasks:1] 2012-03-03 15:41:17,571 StatusLogger.java (line 81) MessagingService n/a 0,0 INFO [CompactionExecutor:37378] 2012-03-03 15:41:17,586 CompactionManager.java (line 608) Compacted to /data/cassandra/data/radarks/RadarEvents-tmp-g-117134-Data.db. 4,541,864 to 4,472,491 (~98% of original) bytes for 2,922 keys. Time: 1,458,339ms. INFO [ScheduledTasks:1] 2012-03-03 15:41:17,598 StatusLogger.java (line 85) ColumnFamily Memtable ops,data Row cache size/cap Key cache size/cap INFO [ScheduledTasks:1] 2012-03-03 15:41:27,619 StatusLogger.java (line 88) system.NodeIdInfo 0,0 0/0 0/0 INFO [CompactionExecutor:37378] 2012-03-03 15:41:27,619 CompactionManager.java (line 542) Compacting Minor: [SSTableReader(path='/data/cassandra/data/OpsCenter/pdps-g-121852-Data.db'), SSTableReader(path='/data/cassandra/data/OpsCenter/pdps-g-121851-Data.db'), SSTableReader(path='/data/cassandra/data/OpsCenter/pdps-g-121856-Data.db'), SSTableReader(path='/data/cassandra/data/OpsCenter/pdps-g-121854-Data.db'), SSTableReader(path='/data/cassandra/data/OpsCenter/pdps-g-121857-Data.db'), SSTableReader(path='/data/cassandra/data/OpsCenter/pdps-g-121853-Data.db'), SSTableReader(path='/data/cassandra/data/OpsCenter/pdps-g-121860-Data.db'), SSTableReader(path='/data/cassandra/data/OpsCenter/pdps-g-121859-Data.db'), SSTableReader(path='/data/cassandra/data/OpsCenter/pdps-g-121858-Data.db'), SSTableReader(path='/data/cassandra/data/OpsCenter/pdps-g-121855-Data.db'), SSTableReader(path='/data/cassandra/data/OpsCenter/pdps-g-121861-Data.db')] INFO [ScheduledTasks:1] 2012-03-03 15:41:27,625 StatusLogger.java (line 88) system.IndexInfo 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,632 StatusLogger.java (line 88) system.LocationInfo 0,0 0/0 1/1 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,632 StatusLogger.java (line 88) system.Migrations 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,632 StatusLogger.java (line 88) system.HintsColumnFamily 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,633 StatusLogger.java (line 88) system.Schema 0,0 0/0 2/2 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,633 StatusLogger.java (line 88) radarks.BatchJobStatus 169,232454 0/0 9/9 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,633 StatusLogger.java (line 88) radarks.RadarEventsByType 2909,775503 0/0 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,633 StatusLogger.java (line 88) radarks.RadarEventsByUser 663,1248692 0/0 35059/35059 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,634 StatusLogger.java (line 88) radarks.RadarEventsScored 2822,1059080 0/0 0/0 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,638 StatusLogger.java (line 88) radarks.RadarEvents 1225,1100147 0/0 14474/14473 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,642 StatusLogger.java (line 88) OpsCenter.rollups60 44,71080 0/0 18/18 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,643 StatusLogger.java (line 88) OpsCenter.pdps 3,4794 0/0 118/118 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,646 StatusLogger.java (line 88) OpsCenter.settings 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,646 StatusLogger.java (line 88) OpsCenter.rollups300 12,19848 0/0 30/30 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,651 StatusLogger.java (line 88) OpsCenter.events 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,653 StatusLogger.java (line 88) OpsCenter.events_timeline 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,660 StatusLogger.java (line 88) OpsCenter.rollups7200 0,0 0/0 10/10 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,668 StatusLogger.java (line 88) OpsCenter.rollups86400 0,0 0/0 0/0 INFO [ScheduledTasks:1] 2012-03-03 15:41:27,692 GCInspector.java (line 122) GC for ConcurrentMarkSweep: 7424 ms for 12 collections, 8369514144 used; max is 8375238656 WARN [ScheduledTasks:1] 2012-03-03 15:41:27,703 GCInspector.java (line 143) Heap is 0.9993164956564075 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically WARN [ScheduledTasks:1] 2012-03-03 15:41:45,757 StorageService.java (line 2418) Flushing ColumnFamilyStore(table='radarks', columnFamily='RadarEventsByType') to relieve memory pressure ERROR [CompactionExecutor:37378] 2012-03-03 15:41:45,764 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:37378,1,main] java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:123) at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.<init>(SSTableWriter.java:560) at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:78) at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:2307) at org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutSizeEstimation(CompactionManager.java:569) at org.apache.cassandra.db.compaction.CompactionManager.doCompaction(CompactionManager.java:507) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:142) at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:108) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) INFO [ScheduledTasks:1] 2012-03-03 15:41:45,794 ColumnFamilyStore.java (line 1128) Enqueuing flush of Memtable-RadarEventsByType@835681722(117741/804828 serialized/live bytes, 3019 ops) ERROR [Thread-568] 2012-03-03 15:41:45,844 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-568,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60) at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:136) ERROR [Thread-555] 2012-03-03 15:41:45,844 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-555,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60) at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:136) ERROR [Thread-560] 2012-03-03 15:41:45,844 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-560,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60) at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490) at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:136) WARN [ScheduledTasks:1] 2012-03-03 15:41:45,850 StorageService.java (line 2422) Flushing ColumnFamilyStore(table='OpsCenter', columnFamily='rollups300') to relieve memory pressure INFO [main] 2012-03-03 15:57:28,265 AbstractCassandraDaemon.java (line 105) Logging initialized INFO [main] 2012-03-03 15:57:28,268 AbstractCassandraDaemon.java (line 126) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_29 INFO [main] 2012-03-03 15:57:28,269 AbstractCassandraDaemon.java (line 127) Heap size: 8375238656/8375238656 INFO [main] 2012-03-03 15:57:28,269 AbstractCassandraDaemon.java (line 128) Classpath: /opt/cassandra/bin/../conf:/opt/cassandra/bin/../build/classes/main:/opt/cassandra/bin/../build/classes/thrift:/opt/cassandra/bin/../lib/antlr-3.2.jar:/opt/cassandra/bin/../lib/apache-cassandra-0.8.6.jar:/opt/cassandra/bin/../lib/apache-cassandra-thrift-0.8.6.jar:/opt/cassandra/bin/../lib/avro-1.4.0-fixes.jar:/opt/cassandra/bin/../lib/avro-1.4.0-sources-fixes.jar:/opt/cassandra/bin/../lib/commons-cli-1.1.jar:/opt/cassandra/bin/../lib/commons-codec-1.2.jar:/opt/cassandra/bin/../lib/commons-collections-3.2.1.jar:/opt/cassandra/bin/../lib/commons-lang-2.4.jar:/opt/cassandra/bin/../lib/concurrentlinkedhashmap-lru-1.1.jar:/opt/cassandra/bin/../lib/guava-r08.jar:/opt/cassandra/bin/../lib/high-scale-lib-1.1.2.jar:/opt/cassandra/bin/../lib/HostNameSubstrSnitch.jar:/opt/cassandra/bin/../lib/jackson-core-asl-1.4.0.jar:/opt/cassandra/bin/../lib/jackson-mapper-asl-1.4.0.jar:/opt/cassandra/bin/../lib/jamm-0.2.2.jar:/opt/cassandra/bin/../lib/jline-0.9.94.jar:/opt/cassandra/bin/../lib/jna.jar:/opt/cassandra/bin/../lib/json-simple-1.1.jar:/opt/cassandra/bin/../lib/libthrift-0.6.jar:/opt/cassandra/bin/../lib/log4j-1.2.16.jar:/opt/cassandra/bin/../lib/mx4j-tools.jar:/opt/cassandra/bin/../lib/servlet-api-2.5-20081211.jar:/opt/cassandra/bin/../lib/slf4j-api-1.6.1.jar:/opt/cassandra/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/cassandra/bin/../lib/snakeyaml-1.6.jar:/opt/cassandra/bin/../lib/jamm-0.2.2.jar INFO [main] 2012-03-03 15:57:30,115 CLibrary.java (line 106) JNA mlockall successful