cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 马超 <hossc...@gmail.com>
Subject Re: High latency of cassandra
Date Tue, 24 Apr 2012 15:56:48 GMT
OK, I wil try ot update the cassandra to 1.0.9 or 1.1.0 and update hector
to 1.0.x

Hope it could rescue me.

Thanks a lot~~

2012/4/24 Ji Cheng <memoleaf@gmail.com>

> The tpstats shows there was no read request pending on this node. Maybe
> you should have a look at the other nodes first.
>
> But my suggestion is to upgrade hector to 1.0.x if possible. Hector 0.7 is
> for cassandra 0.7. Just FYI, we had some issues when upgrading cassandra
> from 0.8 to 1.0. And the problem went away when we upgraded hector as well.
>
> Best,
> Cheng
>
>
>
> 2012/4/24 马超 <hosschao@gmail.com>
>
>> Thank you very much for your replay~
>>
>> The full message like this:
>> DEBUG [Thread-6] 2012-04-24 21:04:11,024 IncomingTcpConnection.java (line
>> 116) Version is now 3
>>
>> During "blocking time" I only saw this message(*appendix shown) *and seems
>> everything is blocked.
>>
>> I logged all cassandra calling time in RPC and there is no large delay
>> but RPC itself delay a lot.
>> I also using JConsole to monitor the GC time (both cassandra and RPC
>> server) and didn't find large GC delay.
>> Is my OS blocked during that time?
>>
>> I also list tpstats as well as compactionstats in the Appendix. Are there
>> something wrong with my cassandra?
>>
>> Thank you very much for your help again :)
>>
>> *Appendix:*
>>
>> *cassandra debug output block:*
>>
>> DEBUG [Thread-6] 2012-04-24 21:04:11,024 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-7] 2012-04-24 21:04:11,430 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-7] 2012-04-24 21:04:11,431 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-4] 2012-04-24 21:04:11,901 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-4] 2012-04-24 21:04:11,902 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-4] 2012-04-24 21:04:12,024 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-7] 2012-04-24 21:04:12,431 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-7] 2012-04-24 21:04:12,431 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:13,024 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:13,750 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:13,751 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-7] 2012-04-24 21:04:14,025 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-4] 2012-04-24 21:04:14,902 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-4] 2012-04-24 21:04:14,902 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:15,025 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:15,751 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:15,751 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:16,025 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-7] 2012-04-24 21:04:16,431 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-7] 2012-04-24 21:04:16,432 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-4] 2012-04-24 21:04:16,902 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-4] 2012-04-24 21:04:16,903 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-7] 2012-04-24 21:04:17,025 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:18,025 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-4] 2012-04-24 21:04:18,902 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-4] 2012-04-24 21:04:18,903 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-7] 2012-04-24 21:04:19,026 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:19,752 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:19,752 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-4] 2012-04-24 21:04:19,903 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-4] 2012-04-24 21:04:19,903 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:20,026 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:20,752 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-6] 2012-04-24 21:04:20,752 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-7] 2012-04-24 21:04:21,026 IncomingTcpConnection.java (line
>> 116) Version is now 3
>> DEBUG [Thread-4] 2012-04-24 21:04:22,026 IncomingTcpConnection.java (line
>> 116) Version is now 3
>>
>> *tpstats:*
>>
>> Pool Name                    Active   Pending      Completed   Blocked
>>  All time blocked
>> ReadStage                         0         0         171002         0
>>               0
>> RequestResponseStage      0         0        1016620         0
>>       0
>> MutationStage                    0         0         278581         0
>>             0
>> ReadRepairStage                0         0              0         0
>>           0
>> ReplicateOnWriteStage        0         0              0         0
>>         0
>> GossipStage                       0         0        1123513         0
>>               0
>> AntiEntropyStage                0         0              0         0
>>             0
>> MigrationStage                    0         0              0         0
>>               0
>> MemtablePostFlusher          0         0             23         0
>>         0
>> StreamStage                       0         0              0         0
>>               0
>> FlushWriter                         0         0             23         0
>>                 0
>> MiscStage                           0         0              0         0
>>                 0
>> InternalResponseStage         0         0              0         0
>>           0
>> HintedHandoff                     0         0              0         0
>>               0
>>
>> Message type           Dropped
>> RANGE_SLICE                  0
>> READ_REPAIR                  0
>> BINARY                             0
>> READ                                0
>> MUTATION                        0
>> REQUEST_RESPONSE     0
>>
>>
>> *compactionstats:*
>>
>> pending tasks: 0
>>
>>
>>
>>
>>
>> 2012/4/24 Maki Watanabe <watanabe.maki@gmail.com>
>>
>>> If you set trace level for IncomingTCPConnection, the message "Version
>>> is now ..." will be printed for every inter-cassandra message received
>>> by the node, including Gossip.
>>> Enabling this log in high traffic will saturate IO for your log disk by
>>> itself.
>>>
>>> You should better to inspect nodetool tpstats, compactionstats at first.
>>>
>>> maki
>>>
>>> 2012/4/24 马超 <hosschao@gmail.com>:
>>> > Does any one have idea for this? Thanks~
>>> >
>>> >
>>> > 2012/4/24 马超 <hosschao@gmail.com>
>>> >>
>>> >> Hi all,
>>> >>
>>> >> I have some troubles of cassandra in my production:
>>> >>
>>> >> I build up a RPC server which using hector client to manipulate the
>>> >> cassandra. Wired things happen nowadays: the latency of RPC sometimes
>>> >> becames very high (10seconds~70seconds) in several minutes and reduce
>>> >> to normal level (30ms in average) after that time. I investigate the
>>> >> debug log of cassandra. During high latency time, the cassandra output
>>> >> lots of message like:
>>> >> "IncomingTcpConnection.java(116) Version is now 3. "
>>> >> Seems everything be blocked during that time.
>>> >>
>>> >> Our settings as following:
>>> >> The version of cassandra is 1.0.1 and hector version is 0.7.0 for the
>>> >> compatible of thrift version which we use (0.5.0)
>>> >> The cluster contains 4 nodes and all of them are seeds. The
>>> >> gcgraceseconds is 0 since we needn't delete the data
>>> >>
>>> >> p.s. It works well for a long time (3 months) but becames crazy these
>>> >> days after we push the new RPC server which supports bigger data
>>> >> saving (2mb in average). I'm not sure if these is the reason.
>>> >>
>>> >> Hope getting your replay~~
>>> >>
>>> >> Thanks,
>>> >>
>>> >> Chao.
>>> >
>>> >
>>>
>>
>>
>

Mime
View raw message