incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maki Watanabe <watanabe.m...@gmail.com>
Subject Re: High latency of cassandra
Date Tue, 24 Apr 2012 10:38:26 GMT
If you set trace level for IncomingTCPConnection, the message "Version
is now ..." will be printed for every inter-cassandra message received
by the node, including Gossip.
Enabling this log in high traffic will saturate IO for your log disk by itself.

You should better to inspect nodetool tpstats, compactionstats at first.

maki

2012/4/24 马超 <hosschao@gmail.com>:
> Does any one have idea for this? Thanks~
>
>
> 2012/4/24 马超 <hosschao@gmail.com>
>>
>> Hi all,
>>
>> I have some troubles of cassandra in my production:
>>
>> I build up a RPC server which using hector client to manipulate the
>> cassandra. Wired things happen nowadays: the latency of RPC sometimes
>> becames very high (10seconds~70seconds) in several minutes and reduce
>> to normal level (30ms in average) after that time. I investigate the
>> debug log of cassandra. During high latency time, the cassandra output
>> lots of message like:
>> "IncomingTcpConnection.java(116) Version is now 3. "
>> Seems everything be blocked during that time.
>>
>> Our settings as following:
>> The version of cassandra is 1.0.1 and hector version is 0.7.0 for the
>> compatible of thrift version which we use (0.5.0)
>> The cluster contains 4 nodes and all of them are seeds. The
>> gcgraceseconds is 0 since we needn't delete the data
>>
>> p.s. It works well for a long time (3 months) but becames crazy these
>> days after we push the new RPC server which supports bigger data
>> saving (2mb in average). I'm not sure if these is the reason.
>>
>> Hope getting your replay~~
>>
>> Thanks,
>>
>> Chao.
>
>

Mime
View raw message