cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joshua Galbraith <jgalbra...@newrelic.com.INVALID>
Subject Re: Too many Cassandra threads waiting!!!
Date Fri, 03 Aug 2018 06:24:59 GMT
Renoy,

Thanks. That kernel is new enough to have the patch for the infamous Linux
kernel futex bug detailed here:
https://groups.google.com/d/topic/mechanical-sympathy/QbmpZxp6C64

To answer your questions above:

What you're seeing is likely just normal behavior for Cassandra and is an
artifact of its staged event driven architecture (SEDA). You can read more
about that if you follow the links in the post above. There is work to move
from SEDA to a thread-per-core (TPC) architecture, which you can read about
in https://issues.apache.org/jira/browse/CASSANDRA-10989.

There are a number of tuning parameters you can tune to adjust the number
of threads working on a few of the various stages within Cassandra (e.g.
memtable_flush_writers, native_transport_max_threads, and
max_hints_delivery_threads).

There will of course be performance impacts for tuning these parameters and
the right values will depend on your data model, hardware, and workload
(among other things).

On Thu, Aug 2, 2018 at 10:44 PM, nokia ceph <nokiacephusers@gmail.com>
wrote:

> Hi Joshua,
>
> # uname -a
> Linux cn6.chn6us1c1.cdn 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27
> UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
> # cat /etc/redhat-release
> CentOS Linux release 7.4.1708 (Core)
>
> On Fri, Aug 3, 2018 at 8:27 AM, Joshua Galbraith <jgalbraith@newrelic.com.
> invalid> wrote:
>
>> Renoy,
>>
>> Out of curiosity, which kernel version are your nodes running?
>>
>> You may find this old message on the mailing list helpful:
>> http://mail-archives.apache.org/mod_mbox/cassandra-user/2016
>> 02.mbox/%3CCAA=6J0-0VabfAn3DJfatOxyJwwEHpdiE67v2wm_
>> u5KAqOrOqnA@mail.gmail.com%3E
>>
>> On Wed, Aug 1, 2018 at 5:38 PM, Elliott Sims <elliott@backblaze.com>
>> wrote:
>>
>>> You might have more luck trying to analyze at the Java level, either via
>>> a (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra
>>> tools like "nodetool tpstats"
>>>
>>> On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph <nokiacephusers@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> i'm having a 5 node cluster with cassandra 3.0.13.
>>>>
>>>> i could see the cassandra process has too many threads.
>>>>
>>>> *# pstree -p `pgrep java` | wc -l*
>>>> *453*
>>>>
>>>> And almost all of those threads are in *sleeping* state and wait at
>>>> *# cat  /proc/166022/task/1698913/wchan*
>>>> *futex_wait_queue_me*
>>>>
>>>> Some more info:
>>>> *# strace -e trace=all -p 166022*
>>>> *strace: Process 166022 attached*
>>>> *futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL*
>>>>
>>>> # cat /proc/166022/stack
>>>> [<ffffffff810f5126>] futex_wait_queue_me+0xc6/0x130
>>>> [<ffffffff810f5deb>] futex_wait+0x17b/0x280
>>>> [<ffffffff810f7b26>] do_futex+0x106/0x5a0
>>>> [<ffffffff810f8040>] SyS_futex+0x80/0x180
>>>> [<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
>>>> [<ffffffffffffffff>] 0xffffffffffffffff
>>>>
>>>>
>>>> What is the reason cassandra is having these many threads? is it the
>>>> normal behavior of cassandra?  Is there a way to reduce this thread count?
>>>> will there be any performance impact because of this (our platform experts
>>>> suspects so)?
>>>>
>>>> Regards,
>>>> Renoy  Paulose
>>>>
>>>>
>>>
>>
>>
>> --
>> *Joshua Galbraith *| Lead Software Engineer | New Relic
>>
>
>


-- 
*Joshua Galbraith *| Lead Software Engineer | New Relic

Mime
View raw message