I have a two machine cluster which is running Cassandra 1.2.6. I am using a keyspace which has a replication factor of 2. But my application demands me to write to both the replicas in parallel and also let the Cassandra do the replication and hoping that Cassandra does not duplicate the key/value on the replica nodes.
- I have nodes Node1 and Node2. I have a keyspace which has replication factor 2 configured on it and a column family to push key/value pairs
- I use a python client (pycassa) to write to the cluster.
- A key, "KeyX", hashes to Node1 and Node2. (I find out which key hashes to which servers through the node tool command. (
$nodetool getendpoints <Keyspace Name> <Column Family Name> <Key Hex String>)
- I use a client to write (KeyX, Value) concurrently to the nodes Node1 and Node2. (In the connection pool I give only the specific server name)
- When writing, I wait for one write to succeed (to the master). (Consistency level ONE)
- Now, I monitor through the
$nodetool statuscommand the amount of disk space that the cluster uses.
- I write around 100 keys each having 2MB values.
Ideally this should store around 400MB on the disk with some over head for storing keys which should be marginal compared to the value size that I using.
- If I do not write to all the nodes that the key hashes to, Cassandra internally handles replication and the data size is around 400MB. (200MB on each node for 100 keys with 2MB value)
- If I do write to all the nodes the key hashes to, Cassandra is writing more than the expected amount of data to the disk. It is as high as 15% more. In our tests Cassandra write ~460MB instead of 400MB.
My question is, is the behavior expected? Is there any configuration that we need to tweak so that Cassandra properly handles concurrent writes to all the replicas.