cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Rose <ianr...@fullstory.com>
Subject Re: significant NICE cpu usage
Date Wed, 08 Oct 2014 18:14:00 GMT
Ah, thanks Andras.

Running `ps -T` I do in fact seen that most Cassandra threads have priority
0 but a few are at priority 4 and these also show non-trivial pcpu.  So
that seems like it!

- Ian


On Wed, Oct 8, 2014 at 11:56 AM, Andras Szerdahelyi <
andras.szerdahelyi@ignitionone.com> wrote:

>  Hello,
>
>  AFAIK Compaction threads run with a lower affinity, I believe that will
> show up as “niced”..
>
>  Regards,
> Andras
>
>   From: Ian Rose <ianrose@fullstory.com>
> Reply-To: user <user@cassandra.apache.org>
> Date: Wednesday 8 October 2014 17:29
> To: user <user@cassandra.apache.org>
> Subject: significant NICE cpu usage
>
>   Hi -
>
>  We are running a small 3-node cassandra cluster on Google Compute
> Engine.  I notice that our machines are reporting (via a collectd agent,
> confirmed by `top`) a significant amount of cpu time in the NICE state.
> For example, one of our machines is a n1-highmem-4 (4 cores, 26 GB RAM).
> Here is the cpu line from to, just now:
>
>  %Cpu(s): 10.3 us, 15.7 sy, 22.8 ni, 44.4 id,  5.3 wa,  0.0 hi,  1.6 si,
>  0.0 st
>
>  So the cpus are spending >20% of the time in NICE, which seems strange.
>
>  Now I have no direct evidence that Cassandra has anything to do with
> this, but we have several other backend services that run on other nodes
> and none of them have shown any significant NICE usage at all.  I have
> tried searching for NICE processes to see if one of them is the source, but
> I can't find anything that looks viable.  For example, this command (which
> I *think* is correct, but please sanity check me) shows that none of the
> processes with a negative priority have noticeable cpu usage.
>
>  $ ps -eo nice,pcpu,pid,args | grep '^\s*\-[1-9]'
> -20  0.0     5 [kworker/0:0H]
> -20  0.0    15 [kworker/1:0H]
> -20  0.0    20 [kworker/2:0H]
> -20  0.0    25 [kworker/3:0H]
> -20  0.0    26 [khelper]
> -20  0.0    28 [netns]
> -20  0.0    29 [writeback]
> -20  0.0    32 [kintegrityd]
> -20  0.0    33 [bioset]
> -20  0.0    34 [crypto]
> -20  0.0    35 [kblockd]
> -20  0.0    47 [kthrotld]
> -20  0.0    48 [ipv6_addrconf]
> -20  0.0    49 [deferwq]
> -20  0.0   162 [scsi_tmf_0]
> -20  0.0   179 [kworker/0:1H]
> -20  0.0   197 [ext4-rsv-conver]
> -20  0.0   214 [kworker/1:1H]
> -20  0.0   480 [kworker/3:1H]
> -20  0.0   481 [kworker/2:1H]
> -20  0.0  1421 [ext4-rsv-conver]
>
>  By comparison, here is the cassandra process (just to verify that pcpu
> is showing real values):
>
>  $ ps -eo nice,pcpu,pid,args | grep cassandra
>   0  217  2498 java -ea
> -javaagent:/mnt/data-1/mn/cassandra/bin/../lib/jamm-0.2.5.jar [...]
>
>  At this point I'm a bit stumped...  any ideas?
>
>  Cheers,
> Ian
>
>

Mime
View raw message