First rule of designing/developing/operating distributed
systems: assume anything and everything can and will happen,
regardless of network configuration or hardware.
This specific situation actually HAS happened to me. Our
Cassandra nodes accept client connections on one ethernet
interface on one network (the production network) yet
communicate with each other on a separate ethernet interface
on a separate network which is Cassandra specific. This was
done mainly due to the relatively large inter-node Cassandra
bandwidth requirements in comparison to client bandwidth
requirements. At one point, the switch for the cassandra
network went down so clients could connect yet the cassandra
nodes could not talk to eachother. (We write at ONE and read
at ALL so everything behaved as expected).