I have some troubles of cassandra in my production:
I build up a RPC server which using hector client to manipulate the
cassandra. Wired things happen nowadays: the latency of RPC sometimes
becames very high (10seconds~70seconds) in several minutes and reduce
to normal level (30ms in average) after that time. I investigate the
debug log of cassandra. During high latency time, the cassandra output
lots of message like:
"IncomingTcpConnection.java(116) Version is now 3. "
Seems everything be blocked during that time.
Our settings as following:
The version of cassandra is 1.0.1 and hector version is 0.7.0 for the
compatible of thrift version which we use (0.5.0)
The cluster contains 4 nodes and all of them are seeds. The
gcgraceseconds is 0 since we needn't delete the data
p.s. It works well for a long time (3 months) but becames crazy these
days after we push the new RPC server which supports bigger data
saving (2mb in average). I'm not sure if these is the reason.
Hope getting your replay~~