cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thunder Stumpges <thunder.stump...@gmail.com>
Subject Re: delay/stall processing reads
Date Thu, 02 Jan 2014 22:24:19 GMT
Thanks Rob, we are using Cassandra 2.0.2, CQL3, native protocol. tpstats is
nearly all zeros from what I can tell. Even running a load of 100rps I can
only ever see 1 or 2 in the active or pending counters, never anything in
the blocked. Even in the "blocked all time" column it is zero in all cases
except for 4 flush writers that have been there for weeks. It is hard for
me to "catch it in the act" since it only lasts maybe 500ms.

Excuse my ignorance, but where would I look for the GC info? What logs
contain this? I will start looking for log files and more clues in them.
Thanks for the quick response!

-Thunder



On Thu, Jan 2, 2014 at 2:15 PM, Robert Coli <rcoli@eventbrite.com> wrote:

> (D'oh, missed your details in the PS.. :D)
>
> I don't know whether the .NET client uses thrift or native protocol..
>
> Re 2.0.2 in production :
> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
>
>
> =Rob
>
>
> On Thu, Jan 2, 2014 at 2:13 PM, Robert Coli <rcoli@eventbrite.com> wrote:
>
>> On Thu, Jan 2, 2014 at 2:05 PM, Thunder Stumpges <
>> thunder.stumpges@gmail.com> wrote:
>>
>>> I am seeing a read operation delay in our small (3 node) cluster where I
>>> am testing. The "normal" latency for these operations is < 2ms as recorded
>>> by our load client. This holds easily beyond several hundred qps. However
>>> there are times when all incoming queries (on a node-by-node basis) are
>>> stalled anywhere from ~100-500ms, and then all "clear" and return at the
>>> same time. This behavior is independent of the amount of load applied; Just
>>> more queries get stalled at higher loads :). It seems like a "stall"
>>> condition happens maybe every 30 seconds or so.
>>>
>>
>> What version of cassandra, what configuration for thrift server if
>> relevant, what protocol being used?
>>
>> The JVM can be expected to pause for 100-500ms while doing GC, cassandra
>> logs the various GC types, what do you see in those logs? What does
>> "nodetool tpstats" say when it happens?
>>
>> =Rob
>>
>>
>
>

Mime
View raw message