cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aoi Kadoya <cadyan....@gmail.com>
Subject Re: CPU high load
Date Thu, 14 Jul 2016 02:36:36 GMT
Hi Romain,

No, I don't think we upgraded cassandra version or changed any of
those schema elements. After I realized this high load issue, I found
that some of the tables have a shorter gc_grace_seconds(1day) than the
rest and because it seemed causing constant compaction cycles, I have
changed them to 10days. but again, that's after load hit this high
number.
some of nodes got eased a little bit after changing gc_grace_seconds
values and repairing nodes, but since few days ago, all of nodes are
constantly reporting load 15-20.

Thank you for the suggestion about logging, let me try to change the
log level to see what I can get from it.

Thanks,
Aoi


2016-07-13 13:28 GMT-07:00 Romain Hardouin <romainh_ml@yahoo.fr>:
> Did you upgrade from a previous version? DId you make some schema changes
> like compaction strategy, compression, bloom filter, etc.?
> What about the R/W requests?
> SharedPool Workers are... shared ;-) Put logs in debug to see some examples
> of what services are using this pool (many actually).
>
> Best,
>
> Romain
>
>
> Le Mercredi 13 juillet 2016 18h15, Patrick McFadin <pmcfadin@gmail.com> a
> écrit :
>
>
> Might be more clear looking at nodetool tpstats
>
> From there you can see all the thread pools and if there are any blocks.
> Could be something subtle like network.
>
> On Tue, Jul 12, 2016 at 3:23 PM, Aoi Kadoya <cadyan.aoi@gmail.com> wrote:
>
> Hi,
>
> I am running 6 nodes vnode cluster with DSE 4.8.1, and since few weeks
> ago, all of the cluster nodes are hitting avg. 15-20 cpu load.
> These nodes are running on VMs(VMware vSphere) that have 8vcpu
> (1core/socket)-16 vRAM.(JVM options : -Xms8G -Xmx8G -Xmn800M)
>
> At first I thought this is because of CPU iowait, however, iowait is
> constantly low(in fact it's 0 almost all time time), CPU steal time is
> also 0%.
>
> When I took a thread dump, I found some of "SharedPool-Worker" threads
> are consuming CPU and those threads seem to be waiting for something
> so I assume this is the cause of cpu load.
>
> "SharedPool-Worker-1" #240 daemon prio=5 os_prio=0
> tid=0x00007fabf459e000 nid=0x39b3 waiting on condition
> [0x00007faad7f02000]
>    java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85)
> at java.lang.Thread.run(Thread.java:745)
>
> Thread dump looks like this, but I am not sure what is this
> sharedpool-worker waiting for.
> Would you please help me with the further trouble shooting?
> I am also reading the thread posted by Yuan as the situation is very
> similar to mine but I didn't get any blocked, dropped or pending count
> in my tpstat result.
>
> Thanks,
> Aoi
>
>
>
>

Mime
View raw message