We have a 10 node cluster (v0.7.9) split into 2 datacenters. Three times we have seen nodes move themselves to different locations in the ring. In each case, the move unbalanced the ring. In one case a node moved to the opposite side of the ring.
Sometime after the first spontaneous move we started using Datastax OpsCenter. The next 2 moves showed up in its event log like:
5/20/2012 11:23am - Info - Host 188.8.131.52 moved from '12345' to '54321'
where '12345' and '54321' are the old and new tokens.
Anyone know whats causing this?