Async Internode Messaging in Apache Cassandra® 4.0

The 4.0 release introduces a couple of protocol optimizations (CASSANDRA-14485). The first removed a bit of redundant information (the IP address and port of the sender, which was included in each message and is now only included once). The second is the use of vint instead of a four-byte int for several values.

Note that Cassandra 4.0 can use these optimizations when communicating between 4.0 nodes, however, 4.0 nodes can also use the old protocol for compatibility with older nodes.

With the introduction of virtual tables in 4.0, it’s easy to get access to messaging metrics. Within the system_views keyspace, there are two new virtual tables named internode_inbound and inernode_outbound. The internode_inbound table contains these metrics:

• Bytes and count of messages that could not be serialized or flushed due to an error
• Bytes and count of messages scheduled
• Bytes and count of messages successfully processed
• Bytes and count of messages successfully received
• Nanoseconds and count of messages throttled
• Bytes and count of messages expired
• Corrupt frames recovered and unrecovered

The inernode_outbound table contains these metrics:

• Bytes and count of messages pending
• Bytes and count of messages sent
• Bytes and count of messages expired
• Bytes and count of messages that could not be sent due to an error
• Bytes and count of messages overloaded
• Active connection count
• Connection attempts
• Successful connection attempts

Hint messaging has been improved in Apache Cassandra® 4.0. As background, hints allow Cassandra’s coordinator nodes to save writes in the case where a replica-owning node is temporarily down. When the replica-owning node comes back up, the coordinator can catch the replica-owning node up by sending hints.

In comparison, when nodes running 4.x need to send hints to other 4.x nodes, they can now use an optimized hint-passing message. This message reduces buffer allocations and eliminates some redundancy.

Paxos refers to an algorithm for establishing consensus within a distributed system. The algorithm has two phases. In the first phase, nodes prepare to accept the most recent state. During the second phase, they propose to commit to that state.

Prior to 4.0, the Paxos prepare and propose messages used the MessagingService stack, even if the message was to the local node. This 4.0 optimization bypasses the MessagingService stack for local messages (CASSANDRA-13862). At first blush, this may not seem like a big deal, but at high-transaction rates, this optimization reduces the P99 latency.