we used to use size tiered compaction then alter column family to leveled compaction.
it seems that each node has a lot of compaction jobs in our current system. (we originally had 4 nodes, recently added 4 so 8 in total now)
e.g. this node has 2143 pending tasks.
pending tasks: 2143
compaction type keyspace column family bytes compacted bytes total progress
Compaction XData UserNameUid 9849441339 20298356110 48.52%
Validation XData UserNameUid 202139799576 758257897889 26.66%
Validation XData UserNameUid 190699406201 760934574422 25.06%
Validation XData UserNameUid 170953615792 758236279998 22.55%
Validation XData UserNameUid 209708959238 761914537718 27.52%
Active compaction remaining time : n/a
however, we see the same column family being in validation in multiple task progress.
The question is that why cassandra seems doing same thing again and again over the same set of data? the column family UserNameUid has 760GB data in total and it seems ridiculous to perform multiple concurrent jobs doing the same thing on the same data.