Hello, I am having an issue where one node of a 2 node cluster seems to be using much more I/O than the other node. Using opscenter, the read/write requests seem to be balanced, but iostat shows the data disk to be maxed at 100% utilization for one machine and <50% for the other. r/s to be about 3x greater on the high i/o node. I am using a RF of 2 and consistency mode of ALL for reads and ONE for writes (current requests are very read heavy). user CPU seems to be fairly low and the same on both machines, but the high i/o machine shows an os load of 34 (!) while the other machine reports 7. I ran a nodetool compactionstats and there are no tasks pending which i assume means there is no compaction going on, and the logs seem to be ok as well. the only difference is that on the high i/o node, i am doing full gc logging, but that's on a separate disk than the data.
Another oddity is that the high i/o node shows a data size of 86GB while the other shows 71GB. I understand there could be differences, but with a RF of 2 I would think they would be roughly the equal?
I am using community version 1.0.10.