Support Forums : DataStax

DataStax Support Forums » Apache Cassandra

Troubles after switching to LeveledCompactionStrategy

(8 posts) (3 voices)

Started 1 year ago by
Latest reply from
This topic is not resolved

Tags:

cheap beats by dre australia

Anonymous
Unregistered

Hi there, I've posted an issue at apache on this (CASSANDRA-4218) - just wandering if there is any experience of it here, excuse the kind of crosspost but its late and I'm running out of options:

After running out of disk space on our six node Cassandra cluster we switched to a Levelled Compaction Strategy. The servers began to degrade performance accruing a large number of pending tasks - we noticed that there was a NullPointerException appearing in the system log:

ERROR [CompactionExecutor:22] 2012-05-03 20:34:38,931 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[CompactionExecutor:22,1,main]
java.lang.NullPointerException
at java.io.File.<init>(File.java:222)
at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:642)
at org.apache.cassandra.db.ColumnFamilyStore.getTempSSTablePath(ColumnFamilyStore.java:653)
at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:1892)
at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:153)
at org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:135)
at org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:115)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Posted 1 year ago #
Anonymous
Unregistered

Hi, it seems like you don't have enough space for compaction to write it's data so getTempSSTablePath gets null into it's directory parameter, that was fixed by CASSANDRA-3985 for C* 1.0.9

Posted 1 year ago #
Anonymous
Unregistered

Thanks, that was what I wondered when I saw the problem - but we have lots of space available on the machine.

Posted 1 year ago #
Anonymous
Unregistered

That could be the case even when you have big free portions of space because there could be minor compactions that would reserve the space and when the next starts it would fail to get some space for itself because all of it was already reserved by others.

Posted 1 year ago #
Anonymous
Unregistered

df reports that there are 368Gb of space available - the database is occupying 434Gb of space - do you think that all of the remainder could be reserved?

Posted 1 year ago #
Anonymous
Unregistered

Take a look at this article http://www.datastax.com/dev/blog/leveled-compaction-in-apache-cassandra, that should help you. Also you can turn on debug logging and see that you get in the "expected data files size is X, largest free partition has Y bytes free" messages.

Posted 1 year ago #
Anonymous
Unregistered

Anyone chasing this problem, there is a productive thread here:

https://issues.apache.org/jira/browse/CASSANDRA-4218

tl;dr: Switching to levelled compaction strategy did not help us escape from disk space hell. It is quite possible that there will be outstanding compaction tasks so large that they will still require all of your disk space. Our resolution was to go back to size tiered compaction, switch on compression and double the number of machines - still creeping towards steady state.

tl;dr: Nearly there.. sleepy.. :)

Thanks for all your advice.

Posted 1 year ago #
Anonymous
Unregistered

cheap dr beats
cheap Beats by dr Dre studio
just beats solo
cheap dr beats
Beats By Dr.Dre Studio
New Style Beats By Dr.Dre
Beats By Dr Dre Studio NFL Pittsburgh Steelers
Beats By Dr Dre Studio NFL San Francisco 49ers

Posted 1 year ago #

RSS feed for this topic