cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdul Patel <abd786...@gmail.com>
Subject Re: All time blocked in nodetool tpstats
Date Thu, 11 Apr 2019 02:57:16 GMT
Do we have any recommendations on concurrents reads ans writes settings?
Mine is 18 node 3 dc cluster with 20 core cpu

On Wednesday, April 10, 2019, Anthony Grasso <anthony.grasso@gmail.com>
wrote:

> Hi Abdul,
>
> Usually we get no noticeable improvement at tuning concurrent_reads and
> concurrent_writes above 128. I generally try to keep current_reads to no
> higher than 64 and concurrent_writes to no higher than 128. In creasing
> the values beyond that you might start running into issues where the kernel
> IO scheduler and/or the disk become saturated. As Paul mentioned, it will
> depend on the size of your nodes though.
>
> If the client is timing out, it is likely that the node that is selected
> as the coordinator for the read has a resource contention somewhere. The
> root cause is usually due to a number of things going on though. As Paul
> mentioned, one of the issues could be the query design. It is worth
> investigating if a particular read query is timing out.
>
> I would also inspect the Cassandra logs and garbage collection logs on the
> node where you are seeing the timeouts. The things to look out for is high
> garbage collection frequency, long garbage collection pauses, and high
> tombstone read warnings.
>
> Regards,
> Anthony
>
> On Thu, 11 Apr 2019 at 06:01, Abdul Patel <abd786.ap@gmail.com> wrote:
>
>> Yes the queries are all select queries as they are more of read intensive
>> app.
>> Last night i rebooted cluster and today they are fine .(i know its
>> temporary) as i still is all time blocked values.
>> I am thinking of incresiing concurrent
>>
>> On Wednesday, April 10, 2019, Paul Chandler <paul@redshots.com> wrote:
>>
>>> Hi Abdul,
>>>
>>> When I have seen dropped messages, I normally double check to ensure the
>>> node not CPU bound.
>>>
>>> If you have a high CPU idle value, then it is likely that tuning the
>>> thread counts will help.
>>>
>>> I normally start with concurrent_reads and concurrent_writes, so in your
>>> case as reads are being dropped then increase concurrent_reads, I normally
>>> change it to 96 to start with, but it will depend on size of your nodes.
>>>
>>> Otherwise it might be badly designed queries, have you investigated
>>> which queries are producing the client timeouts?
>>>
>>> Regards
>>>
>>> Paul Chandler
>>>
>>>
>>>
>>> > On 9 Apr 2019, at 18:58, Abdul Patel <abd786.ap@gmail.com> wrote:
>>> >
>>> > Hi,
>>> >
>>> > My nodetool tpstats arw showing all time blocked high numbers a d also
>>> read dropped messages as 400 .
>>> > Client is expeirince high timeouts.
>>> > Checked few online forums they recommend to increase,
>>> native_transport_max_threads.
>>> > As of jow its commented with 128 ..
>>> > Is it adviabke to increase this and also can this fix timeout issue?
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>
>>>

Mime
View raw message