cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rudi Bruchez <r...@babaluga.com>
Subject Re: timeouts on counter tables
Date Mon, 04 Sep 2017 12:19:56 GMT
It can happen on any of the nodes. We can have a large number of pending 
on ReadStage and CounterMutationStage. We'll try to increase 
concurrent_counter_writes to see how it changes things

> Likely. I believe counter mutations are a tad more expensive than a 
> normal mutation. If you're doing a lot of counter updates that 
> probably doesn't help. Regardless, high amounts of pending 
> reads/mutations is generally not good and indicates the node being 
> overloaded. Are you just seeing this on the 1 node with IO issues or 
> do other nodes have this problem as well?
>
> On 3 September 2017 at 20:25, Rudi Bruchez <rudi@babaluga.com 
> <mailto:rudi@babaluga.com>> wrote:
>
>     Le 30/08/2017 à 05:33, Erick Ramirez a écrit :
>>     Is it possible at all that you may have a data hotspot if it's
>>     not hardware-related?
>>
>>
>     It does not seem so, The partition key seems well distributed and
>     the queries update different keys.
>
>     We have dropped counter_mutation messages in the log :
>
>     COUNTER_MUTATION messages were dropped in last 5000 ms: 0 internal
>     and 2 cross node. Mean internal dropped latency: 0 ms and Mean
>     cross-node dropped latency: 5960 ms
>
>     Pool Name                    Active   Pending Completed   Blocked 
>     All Time Blocked
>     ReadStage    32       503 7481787         0                 0
>     CounterMutationStage     32       221 5722101        
>     0                 0
>
>     The load could be too high ?
>
>     Thanks
>
>


Mime
View raw message