On 6/16/2011 9:36 PM, Dan Hendry wrote:
"Help me out here.  I'm trying to visualize a situation where the clients can access all the C* nodes but the nodes can't access each other.  I don't see how that can happen on a regular ethernet subnet in one data center.  Well, I"m sure there is a case that you can point out.  Ok, I will concede that this is an issue for some network configurations."

First rule of designing/developing/operating distributed systems: assume anything and everything can and will happen, regardless of network configuration or hardware.

This specific situation actually HAS happened to me. Our Cassandra nodes accept client connections on one ethernet interface on one network (the production network) yet communicate with each other on a separate ethernet interface on a separate network which is Cassandra specific. This was done mainly due to the relatively large inter-node Cassandra bandwidth requirements in comparison to client bandwidth requirements. At one point, the switch for the cassandra network went down so clients could connect yet the cassandra nodes could not talk to eachother. (We write at ONE and read at ALL so everything behaved as expected).


Funny, but that's the exact same setup I'm running.  But, I'm not a network guy and kind of assumed it wasn't so typical.  Plus, lately I've had my mind on a cloud setup.