Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C4D2E200B4A for ; Wed, 20 Jul 2016 13:04:32 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id C3273160A73; Wed, 20 Jul 2016 11:04:32 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 979B5160A64 for ; Wed, 20 Jul 2016 13:04:31 +0200 (CEST) Received: (qmail 91614 invoked by uid 500); 20 Jul 2016 11:04:29 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 91604 invoked by uid 99); 20 Jul 2016 11:04:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jul 2016 11:04:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 25C59D0AEE for ; Wed, 20 Jul 2016 11:04:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id qMGwTeuX88Aq for ; Wed, 20 Jul 2016 11:04:25 +0000 (UTC) Received: from mail-vk0-f53.google.com (mail-vk0-f53.google.com [209.85.213.53]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 44A175FD1C for ; Wed, 20 Jul 2016 11:04:25 +0000 (UTC) Received: by mail-vk0-f53.google.com with SMTP id x130so64026562vkc.0 for ; Wed, 20 Jul 2016 04:04:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=sDWfBWbPG49eEgvpbrX9KDxNLiqOwu4UrTPIAgfz5AU=; b=gbBEOaoDERCwOQtkxZEmRe2Llvs+u2rmCIB5x5x0jMy2SVY8C2x+/zM1ErTtJPYBMM hO6BhdEf0MgoONntwcbSJZ5aNWQlqjKnLp1x8QAuJGzL3MLs3hi3+nOlGbbsNsvpydRP jlFzRVgQl+fy0xGyEjCmg3mOTgSwvxR+pDKgH38/lQn1+Aal10NSP0psd5EMNppD7sEI ttXtP2iMChc2n7W7RwmXXjMLQfr5EZfRTb4pEfSfoLUTL89rxM1PLXkZS7N2K1k18lG9 s+VJu34t7Ydbj7B/4Il0B5OTjEQltLQ9ip1R0XoCHu2JZCUqbhAszNj2ZXdxqHRMsAy7 esGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=sDWfBWbPG49eEgvpbrX9KDxNLiqOwu4UrTPIAgfz5AU=; b=R/knn7vRIqjkxm71nmCEQKNGqssFqmRfbibE6Dq5rXzepqxoDBRzog3BpSHlg2sNiI xTwqwJ11SAhCypej9m4oNKiDa7tgH2kYNunYoNA9tmnOME/0M+tcYibpeJGLJVo4SKtk Y21wvZ0ESHH73KEKJIs+iHy+Jjc93NETU/v/zYfqLL+SiS4XUkBYIiHrWMAiLFBFb96c 2Pbi51C4dsH0LAoBtePwhAPdxkft4z8vpk3Oomncj6n5YmcDRs5nTRH7M+89XXJMmd1L sX4bFSNihEBojq0xl+JtX0RGM2mAzCSj4AM5Oo/P2ccJi+Zl37uoJ43UlC04VmPesLb4 867A== X-Gm-Message-State: ALyK8tINMiMhxZ4zkyakeGnOrwiNnbLPeemb2AbF5uzgH/CNLjnpm/b3aTTnJBESC+xtyA8KXyFhnNQvaFKbUQ== X-Received: by 10.176.69.161 with SMTP id u30mr22740928uau.135.1469012652797; Wed, 20 Jul 2016 04:04:12 -0700 (PDT) MIME-Version: 1.0 Received: by 10.176.65.42 with HTTP; Wed, 20 Jul 2016 04:03:53 -0700 (PDT) In-Reply-To: References: <1929681190.4247536.1468441690529.JavaMail.yahoo@mail.yahoo.com> From: Alain RODRIGUEZ Date: Wed, 20 Jul 2016 13:03:53 +0200 Message-ID: Subject: Re: CPU high load To: user@cassandra.apache.org Cc: Romain Hardouin Content-Type: multipart/alternative; boundary=94eb2c11c9605734c605380f2995 archived-at: Wed, 20 Jul 2016 11:04:33 -0000 --94eb2c11c9605734c605380f2995 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Aoi, > since few weeks > ago, all of the cluster nodes are hitting avg. 15-20 cpu load. > These nodes are running on VMs(VMware vSphere) that have 8vcpu > (1core/socket)-16 vRAM.(JVM options : -Xms8G -Xmx8G -Xmn800M) I take my chance, a few ideas / questions below: - What Cassandra version are you running? - How is your GC doing? - Run something like: grep "GC" /var/log/cassandra/system.log - If you have a lot of long CMS pauses you might not be keeping things in the new gen long enough: Xmn800M looks too small to me, it = has been a default but I never saw a case where this setting worked better than a higher value (let's say 2G), also tenuring threshold gives better results if set a bit higher than default (let's say 16). Those options are in cassandra-env.sh. - Do you have other warnings or errors? Anything about tombstones or compacting wide rows incrementally? - What compaction strategy are you using - How many concurrent compactors do you use (if you have 8 cores, this value should probably be between 2 and 6, 4 is a good starting point) - If your compaction is not fast enough and disk are doing fine, consider increasing the compaction throughput from default 16 to 32 or 6= 4 Mbps to mitigate the impact of the point above. - Do you use compression ? What kind ? - Did the request count increased recently? Do you consider adding capacity or do you think you're hitting a new bug / issue that is worth = it investigating / solving? - Are you using default configuration? What did you change? No matter what you try, do it as much as possible on one canary node first, and incrementally (one change at the time - using NEWHEAP =3D 2GB + tenuringThreshold =3D 16 would be one change, it makes sense to move those = 2 values together) > I have enabled a auto repair service on opscenter and it's running behind Also when did you do that, starting repairs? Repair is an expensive operation, consuming a lot of resources that is often needed, but that is hard to tune correctly. Are you sure you have enough CPU power to handle the load + repairs? Some other comments probably not directly related: > I also realized that my cluster isn't well balanced Well you cluster looks balanced to me 7 GB isn't that far from 11 GB. To have a more accurate information, use 'nodetool status mykeyspace'. This way ownership will be displayed, replacing (?) by ownership (xx %). Total ownership =3D 300 % in your case (RF=3D3) > I am running 6 nodes vnode cluster with DSE 4.8.1, and since few weeks > ago, all of the cluster nodes are hitting avg. 15-20 cpu load. By the way, from https://docs.datastax.com/en/datastax_enterprise/4.8/datastax_enterprise/RN= dse.html : "Warning: DataStax does not recommend 4.8.1 or 4.8.2 versions for production, see warning. Use 4.8.3 instead.". I am not sure what happened there but I would move to 4.8.3+ asap, datastax people know their products and I don't like this kind of orange and bold warnings :-). C*heers, ----------------------- Alain Rodriguez - alain@thelastpickle.com France The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com 2016-07-14 4:36 GMT+02:00 Aoi Kadoya : > Hi Romain, > > No, I don't think we upgraded cassandra version or changed any of > those schema elements. After I realized this high load issue, I found > that some of the tables have a shorter gc_grace_seconds(1day) than the > rest and because it seemed causing constant compaction cycles, I have > changed them to 10days. but again, that's after load hit this high > number. > some of nodes got eased a little bit after changing gc_grace_seconds > values and repairing nodes, but since few days ago, all of nodes are > constantly reporting load 15-20. > > Thank you for the suggestion about logging, let me try to change the > log level to see what I can get from it. > > Thanks, > Aoi > > > 2016-07-13 13:28 GMT-07:00 Romain Hardouin : > > Did you upgrade from a previous version? DId you make some schema chang= es > > like compaction strategy, compression, bloom filter, etc.? > > What about the R/W requests? > > SharedPool Workers are... shared ;-) Put logs in debug to see some > examples > > of what services are using this pool (many actually). > > > > Best, > > > > Romain > > > > > > Le Mercredi 13 juillet 2016 18h15, Patrick McFadin > a > > =C3=A9crit : > > > > > > Might be more clear looking at nodetool tpstats > > > > From there you can see all the thread pools and if there are any blocks= . > > Could be something subtle like network. > > > > On Tue, Jul 12, 2016 at 3:23 PM, Aoi Kadoya > wrote: > > > > Hi, > > > > I am running 6 nodes vnode cluster with DSE 4.8.1, and since few weeks > > ago, all of the cluster nodes are hitting avg. 15-20 cpu load. > > These nodes are running on VMs(VMware vSphere) that have 8vcpu > > (1core/socket)-16 vRAM.(JVM options : -Xms8G -Xmx8G -Xmn800M) > > > > At first I thought this is because of CPU iowait, however, iowait is > > constantly low(in fact it's 0 almost all time time), CPU steal time is > > also 0%. > > > > When I took a thread dump, I found some of "SharedPool-Worker" threads > > are consuming CPU and those threads seem to be waiting for something > > so I assume this is the cause of cpu load. > > > > "SharedPool-Worker-1" #240 daemon prio=3D5 os_prio=3D0 > > tid=3D0x00007fabf459e000 nid=3D0x39b3 waiting on condition > > [0x00007faad7f02000] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > > at java.lang.Thread.run(Thread.java:745) > > > > Thread dump looks like this, but I am not sure what is this > > sharedpool-worker waiting for. > > Would you please help me with the further trouble shooting? > > I am also reading the thread posted by Yuan as the situation is very > > similar to mine but I didn't get any blocked, dropped or pending count > > in my tpstat result. > > > > Thanks, > > Aoi > > > > > > > > > --94eb2c11c9605734c605380f2995 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Aoi,
=C2=A0
since few weeks
ago, all of the cluster nodes are hit= ting avg. 15-20 cpu load.
These nodes are running on VMs(VMware vSphere) that h= ave 8vcpu
(1core/socket)-16 vRAM.(JVM options : -Xms8G -Xmx8G -Xmn800M)

I take my chance, a few ideas / questions be= low: =C2=A0
  • What Cassandra version are you running?
    • <= li>How is your GC doing?=C2=A0=
    • Run something like: grep "GC&= quot; /var/log/cassandra/system.log
    • If you have a lot of long CMS pauses you might not be keeping thin= gs in the new gen long enough: Xmn800M looks too small to me, it has been a= default but I never saw a case where this setting worked better than a hig= her value (let's say 2G), also = tenuring threshold gives better results if set a bit higher than default (l= et's say 16). Those options are in cassandra-env.sh.
    Do you have other warnings or errors? An= ything about tombstones or compacting wide rows incrementally?<= li>What compaction strategy are you using<= /span>
  • How many concurrent compact= ors do you use (if you have 8 cores, this value should probably be between = 2 and 6, 4 is a good starting point)
  • If your compaction is not fast enough and disk are doing fine, co= nsider increasing the compaction throughput from default 16 to 32 or 64 Mbp= s to mitigate the impact of the point above.
  • Do you use compression ? What kind ?
  • Did the request count increased recently? Do yo= u consider adding capacity or do you think you're hitting a new bug / i= ssue that is worth it investigating / solving?
  • Are you using default configuration? What did you cha= nge?
No matter w= hat you try, do it as much as possible on one canary node first, and increm= entally (one change at the time - using NEWHEAP =3D 2GB + tenuringThreshold= =3D 16 would be one change, it makes sense to move those 2 values together= )
=C2=A0
I have enabled a auto repair service on opscenter and it's = running=C2=A0behind
=C2=A0
Also when did yo= u do that, starting repairs? Repair is an expensive operation, consuming a = lot of resources that is often needed, but that is hard to tune correctly. = Are you sure you have enough CPU power to handle the load + repairs?=

Some other comments probably not directly r= elated:
=C2=A0
I also realized that my cluster isn't well balanced

Well you cluster looks balanced to me 7 GB isn'= ;t that far from 11 GB. To have a more accurate information, use 'nodet= ool status mykeyspace'. This way ownership will be displayed, replacing= (?) by ownership (xx %). Total ownership =3D 300 % in your case (RF=3D3)
=C2=A0
I am= running 6 nodes vnode cluster with DSE 4.8.1, and since few weeksago, all of t= he cluster nodes are hitting avg. 15-20 cpu load.
<= br>
By the way, from https://docs.datastax.c= om/en/datastax_enterprise/4.8/datastax_enterprise/RNdse.html:=C2=A0

"Warning:=C2=A0DataStax does not recom= mend 4.8.1 or 4.8.2 versions for production, see warning. Use 4.8.3 instead= .".

I am not sure what happened there = but I would move to 4.8.3+ asap, datastax people know their products and I = don't like this kind of orange and bold warnings :-).

C*heers,
-----------------------
Alain Rod= riguez - alain@thelastpickle.com=
France

The Last Pickle - Apache Cas= sandra Consulting

=
2016-07-14 4:36 GMT+02:00 Aoi Kadoya <cady= an.aoi@gmail.com>:
Hi Romai= n,

No, I don't think we upgraded cassandra version or changed any of
those schema elements. After I realized this high load issue, I found
that some of the tables have a shorter gc_grace_seconds(1day) than the
rest and because it seemed causing constant compaction cycles, I have
changed them to 10days. but again, that's after load hit this high
number.
some of nodes got eased a little bit after changing gc_grace_seconds
values and repairing nodes, but since few days ago, all of nodes are
constantly reporting load 15-20.

Thank you for the suggestion about logging, let me try to change the
log level to see what I can get from it.

Thanks,
Aoi


2016-07-13 13:28 GMT-07:00 Romain Hardouin <romainh_ml@yahoo.fr>:
> Did you upgrade from a previous version? DId you make some schema chan= ges
> like compaction strategy, compression, bloom filter, etc.?
> What about the R/W requests?
> SharedPool Workers are... shared ;-) Put logs in debug to see some exa= mples
> of what services are using this pool (many actually).
>
> Best,
>
> Romain
>
>
> Le Mercredi 13 juillet 2016 18h15, Patrick McFadin <pmcfadin@gmail.com> a
> =C3=A9crit :
>
>
> Might be more clear looking at nodetool tpstats
>
> From there you can see all the thread pools and if there are any block= s.
> Could be something subtle like network.
>
> On Tue, Jul 12, 2016 at 3:23 PM, Aoi Kadoya <cadyan.aoi@gmail.com> wrote:
>
> Hi,
>
> I am running 6 nodes vnode cluster with DSE 4.8.1, and since few weeks=
> ago, all of the cluster nodes are hitting avg. 15-20 cpu load.
> These nodes are running on VMs(VMware vSphere) that have 8vcpu
> (1core/socket)-16 vRAM.(JVM options : -Xms8G -Xmx8G -Xmn800M)
>
> At first I thought this is because of CPU iowait, however, iowait is > constantly low(in fact it's 0 almost all time time), CPU steal tim= e is
> also 0%.
>
> When I took a thread dump, I found some of "SharedPool-Worker&quo= t; threads
> are consuming CPU and those threads seem to be waiting for something > so I assume this is the cause of cpu load.
>
> "SharedPool-Worker-1" #240 daemon prio=3D5 os_prio=3D0
> tid=3D0x00007fabf459e000 nid=3D0x39b3 waiting on condition
> [0x00007faad7f02000]
>=C2=A0 =C2=A0 java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:85) > at java.lang.Thread.run(Thread.java:745)
>
> Thread dump looks like this, but I am not sure what is this
> sharedpool-worker waiting for.
> Would you please help me with the further trouble shooting?
> I am also reading the thread posted by Yuan as the situation is very > similar to mine but I didn't get any blocked, dropped or pending c= ount
> in my tpstat result.
>
> Thanks,
> Aoi
>
>
>
>

--94eb2c11c9605734c605380f2995--