not resolved
Anonymous
Unregistered
Hi,
On this documentation page from Datastax (http://www.datastax.com/docs/1.0/operations/tuning) they suggests to NOT run major compaction (all the way at the bottom), however elsewhere on the net i read that people still do major compaction.
Perhaps somebody can explain when to not run it and when you should?
thanks!
Hello Walt,
Most times those who run major compactions are doing so since they had already run the first major compaction and need to continue doing so.
If you have something like a queue application, which generates many tombstones, with SSDs, then running major compaction isn't much of an issue and thus recommended.
If your workload generates many tombstones, then major compaction may be an option, provided you can handle the IO. But in most cases you're better off without major compactions and instead properly tuning your compaction:
http://www.datastax.com/docs/1.0/operations/tuning#tuning-compaction
Thanks,
Joaquin
Anonymous
Unregistered
Thanks Joaquin,
just a follow up:
We did do one major compaction on one of our nodes in the past, would we be able to so to speak 'reset' it? perhaps by doing a full data reload on that node?
Anonymous
Unregistered
"Also, once you run a major compaction, automatic minor compactions are no longer triggered frequently forcing you to manually run major compactions on a routine basis."
does anybody know how to undo this ?
No, I'm sorry, once a major compaction is run, you have to stick with performing them. Typically once per gc_grace_period.
You should probably look into leveled compaction if you are on 1.0, this should help relieve some of the stress and process of having to run major compactions.
http://www.datastax.com/dev/blog/leveled-compaction-in-apache-cassandra
Thanks,
Joaquin
Anonymous
Unregistered
thanks again Joaquin !
hmm thats disappointing... so even a complete emptying of the data directory will not work?
What will happen if we don't do those major compactions?
In our setup, i can see that the node that had one major compaction in the past it is doing normal compactions sometimes...
Not a problem Fransen.
More info on compaction can be found here:
http://www.datastax.com/docs/1.0/dml/about_writes#about-compaction
Without major compactions, that big sstable won't go through any of those changes.
If you won't be reading much from that keyspace and are noticing minor compaction still occurring then you should relatively not have to worry.
If you wish to empty out the entire data directory and nuke all the previous information, this is fine too as long as you understand you lose that data. If that's the case, then the record of major compactions will be ultimately deleted.
Thanks,
Joaquin
It isn't that minor compactions are run less frequently because of the major compaction per se, but because major compaction creates One Big Data File by definition, it will be some time before new data files get large enough to be eligible to be compacted with it. Newly flushed data files will thus continue to be compacted with each other normally.
If you're finding that normal minor compactions aren't doing a good enough job of merging together overwritten row versions, you should give the leveled compaction strategy a try. I wrote more about compaction in general and leveled in particular here: http://www.datastax.com/dev/blog/leveled-compaction-in-apache-cassandra
Anonymous
Unregistered
Thank you both for your answers, much appreciated!
I don't think leveled compaction is an improvement for us. we do mostly writes and are not concerned with duplicate data. Our biggest worry at the moment are minor compactions on CF's that are quite big (100GB). they make the node response very slow to our reads.
Any tips on solving slow reads if a node is doing (minor) compaction on a big CF ?
thanks a bunch.
Fransen
