cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nokia ceph <nokiacephus...@gmail.com>
Subject Re: Too many Cassandra threads waiting!!!
Date Fri, 03 Aug 2018 05:44:46 GMT
Hi Joshua,

# uname -a
Linux cn6.chn6us1c1.cdn 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27
UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

On Fri, Aug 3, 2018 at 8:27 AM, Joshua Galbraith <
jgalbraith@newrelic.com.invalid> wrote:

> Renoy,
>
> Out of curiosity, which kernel version are your nodes running?
>
> You may find this old message on the mailing list helpful:
> http://mail-archives.apache.org/mod_mbox/cassandra-user/
> 201602.mbox/%3CCAA=6J0-0VabfAn3DJfatOxyJwwEHpdiE67v2w
> m_u5KAqOrOqnA@mail.gmail.com%3E
>
> On Wed, Aug 1, 2018 at 5:38 PM, Elliott Sims <elliott@backblaze.com>
> wrote:
>
>> You might have more luck trying to analyze at the Java level, either via
>> a (Java) stack dump and the "ttop" tool from Swiss Java Knife, or Cassandra
>> tools like "nodetool tpstats"
>>
>> On Wed, Aug 1, 2018 at 2:08 AM, nokia ceph <nokiacephusers@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> i'm having a 5 node cluster with cassandra 3.0.13.
>>>
>>> i could see the cassandra process has too many threads.
>>>
>>> *# pstree -p `pgrep java` | wc -l*
>>> *453*
>>>
>>> And almost all of those threads are in *sleeping* state and wait at
>>> *# cat  /proc/166022/task/1698913/wchan*
>>> *futex_wait_queue_me*
>>>
>>> Some more info:
>>> *# strace -e trace=all -p 166022*
>>> *strace: Process 166022 attached*
>>> *futex(0x7efc24aeb9d0, FUTEX_WAIT, 166023, NULL*
>>>
>>> # cat /proc/166022/stack
>>> [<ffffffff810f5126>] futex_wait_queue_me+0xc6/0x130
>>> [<ffffffff810f5deb>] futex_wait+0x17b/0x280
>>> [<ffffffff810f7b26>] do_futex+0x106/0x5a0
>>> [<ffffffff810f8040>] SyS_futex+0x80/0x180
>>> [<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
>>> [<ffffffffffffffff>] 0xffffffffffffffff
>>>
>>>
>>> What is the reason cassandra is having these many threads? is it the
>>> normal behavior of cassandra?  Is there a way to reduce this thread count?
>>> will there be any performance impact because of this (our platform experts
>>> suspects so)?
>>>
>>> Regards,
>>> Renoy  Paulose
>>>
>>>
>>
>
>
> --
> *Joshua Galbraith *| Lead Software Engineer | New Relic
>

Mime
View raw message