Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5D124106D7 for ; Wed, 19 Feb 2014 18:02:45 +0000 (UTC) Received: (qmail 72721 invoked by uid 500); 19 Feb 2014 18:02:40 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 72701 invoked by uid 500); 19 Feb 2014 18:02:40 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 72693 invoked by uid 99); 19 Feb 2014 18:02:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Feb 2014 18:02:40 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ynerella999@gmail.com designates 209.85.160.46 as permitted sender) Received: from [209.85.160.46] (HELO mail-pb0-f46.google.com) (209.85.160.46) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Feb 2014 18:02:36 +0000 Received: by mail-pb0-f46.google.com with SMTP id um1so720345pbc.19 for ; Wed, 19 Feb 2014 10:02:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=i3LY90SaNrzxFFvRds0l7Re7SkWivmcGwR6nsmH2MLg=; b=JPlh55KkxzOnDNzh3xRwx3VF3L4hU/Ex0Hk7tIibMe5Nmd7vnFpwYY2Yor0Acgd4HS HIb/mcSm/XnR+AQwaLRH6xUL1G6MlnFbolUMBEK+DCFwwuP1dYkMiwQBTaTZJMqsnrDL RU/MV6gnJiIi2MgZ/Wa6WB+Pyyy0Lo2cu9MJKFF7XlQJgk9qUjYaDbMEasCBEmcSNSoC kwjZ7hrw1mDsfi2UUmOZus0RFVgZMdq8PPZwlnpAXwPyR8tvoBiY9eGkhxGvbpeVKePp nwLsZQPBoxJnzxxUl3Qpyn9YQo6fQ2URQS+FVGTD7YzLqzNKluTg3t9VX7R5wzksV6Om jPMw== MIME-Version: 1.0 X-Received: by 10.66.136.103 with SMTP id pz7mr3838935pab.140.1392832935270; Wed, 19 Feb 2014 10:02:15 -0800 (PST) Received: by 10.68.19.201 with HTTP; Wed, 19 Feb 2014 10:02:15 -0800 (PST) In-Reply-To: References: Date: Wed, 19 Feb 2014 10:02:15 -0800 Message-ID: Subject: Re: High CPU load on one node in the cluster From: Yogi Nerella To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11332b6c569ef004f2c630a6 X-Virus-Checked: Checked by ClamAV on apache.org --001a11332b6c569ef004f2c630a6 Content-Type: text/plain; charset=ISO-8859-1 You should start your Cassandra daemon with -verbose:gc (please check syntax) and then run it in foreground, as Cassandra closes the standard out) Please see other emails in this forum for getting Garbage Collection Statistics from Cassandra user mail, or look at any Java specific sites. Ex: http://stackoverflow.com/questions/1161647/how-to-redirect-verbose-garbage-collection-output-to-a-file It depends on what JVM you are running. On Wed, Feb 19, 2014 at 9:12 AM, Sourabh Agrawal wrote: > How do I get that statistic? > > > On Wed, Feb 19, 2014 at 10:34 PM, Yogi Nerella wrote: > >> Could be your -Xmn800M is too low, that is why it is trying garbage >> collecting very frequently. >> Do you have any statistics on how much memory it is collecting on every >> cycle? >> >> >> >> On Wed, Feb 19, 2014 at 8:47 AM, Sourabh Agrawal wrote: >> >>> Below is CPU usage from top. I don't see any steal. Idle time is pretty >>> low. >>> >>> Cpu(s): 83.3%us, 14.5%sy, 0.0%ni, 0.5%id, 0.0%wa, 0.0%hi, 1.7%si, >>> 0.0%st >>> >>> Any other pointers? >>> >>> >>> On Wed, Feb 19, 2014 at 8:34 PM, Nate McCall wrote: >>> >>>> You may be seeing steal from another tenant on the VM. This article has >>>> a good explanation: >>>> >>>> http://blog.scoutapp.com/articles/2013/07/25/understanding-cpu-steal-time-when-should-you-be-worried >>>> >>>> In short, kill the instance and launch a new one. Depending on your >>>> latency requirements and operational ability to respond, you may want to >>>> consider paying for dedicated instances. >>>> >>>> >>>> On Wed, Feb 19, 2014 at 2:30 AM, Sourabh Agrawal < >>>> iitr.sourabh@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am running cassandra 2.0.3 cluster on 4 AWS nodes. memory arguments >>>>> are the following for each node : >>>>> -Xms8G -Xmx8G -Xmn800M >>>>> >>>>> I am experiencing consistent high loads on one of the nodes. Each node >>>>> is getting approximately equal number of writes. I tried to have a look at >>>>> the logs and seems like CMS GC is running every 1-2 seconds. >>>>> >>>>> Any pointers on how to debug this? >>>>> >>>>> -- >>>>> Sourabh Agrawal >>>>> Bangalore >>>>> +91 9945657973 >>>>> >>>> >>>> >>>> >>>> -- >>>> ----------------- >>>> Nate McCall >>>> Austin, TX >>>> @zznate >>>> >>>> Co-Founder & Sr. Technical Consultant >>>> Apache Cassandra Consulting >>>> http://www.thelastpickle.com >>>> >>> >>> >>> >>> -- >>> Sourabh Agrawal >>> Bangalore >>> +91 9945657973 >>> >> >> > > > -- > Sourabh Agrawal > Bangalore > +91 9945657973 > --001a11332b6c569ef004f2c630a6 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
You should start your Cassandra daemon with -verbose:gc (p= lease check syntax) and then run it in foreground, as Cassandra closes the = standard out)
Please see other emails in this forum for getting Garbage= Collection Statistics from Cassandra user mail, or look at any Java specif= ic sites.


<= br>
On Wed, Feb 19, 2014 at 9:12 AM, Sourabh Agra= wal <iitr.sourabh@gmail.com> wrote:
How do I get that statistic= ?
<= br>
On Wed, Feb 19, 2014 at 10:34 PM, Yogi Nerella <ynerella999@gmail.com<= /a>> wrote:
Could be your -Xmn800M is t= oo low, that is why it is trying garbage collecting very frequently. =A0=A0=
Do you have any statistics on how much memory it is collecting on every cyc= le? =A0=A0



On Wed, Feb 1= 9, 2014 at 8:47 AM, Sourabh Agrawal <iitr.sourabh@gmail.com> wrote:
Below is CPU usage fro= m top. I don't see any steal. Idle time is pretty low.

Cpu(s): 83.3%us, 14.5%sy, =A00.0%ni, =A00.5%id, =A00.0%wa, =A00.0%hi, =A01.= 7%si, =A00.0%st

Any other pointers?

On Wed, Feb 19, 2014 at 8:34 PM, Nate McCall <nate@thelastpickle.com> wrote:
You may be seeing steal fro= m another tenant on the VM. This article has a good explanation:

In short, kill the instance and launch a new one.= Depending on your latency requirements and operational ability to respond,= you may want to consider paying for dedicated instances.


On Wed, Feb 19, 2014 at 2:30 AM, Sourabh= Agrawal <iitr.sourabh@gmail.com> wrote:
Hi,

I am running cassandra 2.0.3 cluste= r on 4 AWS nodes. memory arguments are the following for each node :=A0
-Xms8G -Xmx8G -Xmn800M

I am experiencin= g consistent high loads on one of the nodes. Each node is getting approxima= tely equal number of writes. I tried to have a look at the logs and seems l= ike CMS GC is running every 1-2 seconds.

Any pointers on how to debug this?

--
Sourabh Agrawal<= div>Bangalore



<= font color=3D"#888888">--
-----------------
Nate Mc= Call
Austin, TX
@zznate

Co-Founder & Sr. Technical Consult= ant
Apache Cassandra Consulting
http://www.thelastpickle.com



--
Sourabh Agrawal
Bangalore




--
=
Sourabh Agrawal
Bangalore

--001a11332b6c569ef004f2c630a6--