Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 16C7A1136C for ; Thu, 20 Feb 2014 14:03:54 +0000 (UTC) Received: (qmail 31196 invoked by uid 500); 20 Feb 2014 14:03:50 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 30693 invoked by uid 500); 20 Feb 2014 14:03:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 30584 invoked by uid 99); 20 Feb 2014 14:03:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Feb 2014 14:03:48 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of edlinuxguru@gmail.com designates 74.125.82.49 as permitted sender) Received: from [74.125.82.49] (HELO mail-wg0-f49.google.com) (74.125.82.49) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Feb 2014 14:03:42 +0000 Received: by mail-wg0-f49.google.com with SMTP id y10so1467967wgg.28 for ; Thu, 20 Feb 2014 06:03:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=pmQa/8Yv7ulG/RmpUb0Vw/KKhTMR8GOCgVKG5OzblbM=; b=Q5Hm0qjY8cPCLI9g6nAis90ewKPZI1u8lakhPNZRv9t8n6dXzTXifgl+qOwlDXL/Gf z3pNpnz4/NZB22wfIHiKcxtfHzjKRA5QGppb4qw0JPsmervPXXEbcA6yViV9ck230jr2 7yIw5WqDOHdeXScpoqCojMA7at4VWxR0x/vf8Tlt9+SZoe1yLd8eL8dOASlP90PONti7 zXiOyk+rPva79tC4d8PpWkFzxCyHtNAzl9nB0qNGhXRni1/d9knOgk5s1ScxtJj7819l zpJpEn2YtdgAIeZtLRsDhK5oDslM1ZzdToRl/HWLmZG2eDqlTx4Re9JHpq3WFCauLfyX yg6g== MIME-Version: 1.0 X-Received: by 10.194.2.110 with SMTP id 14mr2139978wjt.96.1392905002649; Thu, 20 Feb 2014 06:03:22 -0800 (PST) Received: by 10.194.220.105 with HTTP; Thu, 20 Feb 2014 06:03:22 -0800 (PST) In-Reply-To: References: Date: Thu, 20 Feb 2014 09:03:22 -0500 Message-ID: Subject: Re: High CPU load on one node in the cluster From: Edward Capriolo To: "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary=047d7b33db86e38fff04f2d6f736 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b33db86e38fff04f2d6f736 Content-Type: text/plain; charset=ISO-8859-1 Upgrade from 2.0.3. There are several bugs, On Wednesday, February 19, 2014, Yogi Nerella wrote: > You should start your Cassandra daemon with -verbose:gc (please check syntax) and then run it in foreground, as Cassandra closes the standard out) > Please see other emails in this forum for getting Garbage Collection Statistics from Cassandra user mail, or look at any Java specific sites. > Ex: http://stackoverflow.com/questions/1161647/how-to-redirect-verbose-garbage-collection-output-to-a-file > > > It depends on what JVM you are running. > > > On Wed, Feb 19, 2014 at 9:12 AM, Sourabh Agrawal wrote: >> >> How do I get that statistic? >> >> On Wed, Feb 19, 2014 at 10:34 PM, Yogi Nerella wrote: >>> >>> Could be your -Xmn800M is too low, that is why it is trying garbage collecting very frequently. >>> Do you have any statistics on how much memory it is collecting on every cycle? >>> >>> >>> On Wed, Feb 19, 2014 at 8:47 AM, Sourabh Agrawal wrote: >>>> >>>> Below is CPU usage from top. I don't see any steal. Idle time is pretty low. >>>> Cpu(s): 83.3%us, 14.5%sy, 0.0%ni, 0.5%id, 0.0%wa, 0.0%hi, 1.7%si, 0.0%st >>>> >>>> Any other pointers? >>>> >>>> On Wed, Feb 19, 2014 at 8:34 PM, Nate McCall wrote: >>>>> >>>>> You may be seeing steal from another tenant on the VM. This article has a good explanation: >>>>> http://blog.scoutapp.com/articles/2013/07/25/understanding-cpu-steal-time-when-should-you-be-worried >>>>> >>>>> In short, kill the instance and launch a new one. Depending on your latency requirements and operational ability to respond, you may want to consider paying for dedicated instances. >>>>> >>>>> On Wed, Feb 19, 2014 at 2:30 AM, Sourabh Agrawal < iitr.sourabh@gmail.com> wrote: >>>>>> >>>>>> Hi, >>>>>> I am running cassandra 2.0.3 cluster on 4 AWS nodes. memory arguments are the following for each node : >>>>>> -Xms8G -Xmx8G -Xmn800M >>>>>> >>>>>> I am experiencing consistent high loads on one of the nodes. Each node is getting approximately equal number of writes. I tried to have a look at the logs and seems like CMS GC is running every 1-2 seconds. >>>>>> Any pointers on how to debug this? >>>>>> -- >>>>>> Sourabh Agrawal >>>>>> Bangalore >>>>>> +91 9945657973 >>>>> >>>>> >>>>> -- >>>>> ----------------- >>>>> Nate McCall >>>>> Austin, TX >>>>> @zznate >>>>> >>>>> Co-Founder & Sr. Technical Consultant >>>>> Apache Cassandra Consulting >>>>> http://www.thelastpickle.com >>>> >>>> >>>> -- >>>> Sourabh Agrawal >>>> Bangalore >>>> +91 9945657973 >> >> >> >> -- >> Sourabh Agrawal >> Bangalore >> +91 9945657973 > -- Sorry this was sent from mobile. Will do less grammar and spell check than usual. --047d7b33db86e38fff04f2d6f736 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Upgrade from 2.0.3. There are several bugs,

On Wednesday, February 1= 9, 2014, Yogi Nerella <ynerella= 999@gmail.com> wrote:
> You should start your Cassandra daemon= with -verbose:gc (please check syntax) and then run it in foreground, as C= assandra closes the standard out)
> Please see other emails in this forum for getting Garbage Collection S= tatistics from Cassandra user mail, or look at any Java specific sites.
= > Ex: =A0http://stackoverflow.co= m/questions/1161647/how-to-redirect-verbose-garbage-collection-output-to-a-= file
>
>
> It depends on what JVM you are running.
>
>= ;
> On Wed, Feb 19, 2014 at 9:12 AM, Sourabh Agrawal <iitr.sourabh@gmail.com> wrote:
>= >
>> How do I get that statistic?
>>
>> On Wed, Feb 1= 9, 2014 at 10:34 PM, Yogi Nerella <ynerella999@gmail.com> wrote:
>>>
>>> Co= uld be your -Xmn800M is too low, that is why it is trying garbage collectin= g very frequently. =A0=A0
>>> Do you have any statistics on how much memory it is collecting= on every cycle? =A0=A0
>>>
>>>
>>> On = Wed, Feb 19, 2014 at 8:47 AM, Sourabh Agrawal <iitr.sourabh@gmail.com> wrote:
>>>>
>>>> Below is CPU usage from top. I don'= ;t see any steal. Idle time is pretty low.
>>>> Cpu(s): 83.3= %us, 14.5%sy, =A00.0%ni, =A00.5%id, =A00.0%wa, =A00.0%hi, =A01.7%si, =A00.0= %st
>>>>
>>>> Any other pointers?
>>>>
>>>>= ; On Wed, Feb 19, 2014 at 8:34 PM, Nate McCall <nate@thelastpickle.com> wrote:
>>>>= >
>>>>> You may be seeing steal from another tenant on the VM.= This article has a good explanation:
>>>>> http://blog.scoutapp.com/articles/2013/07/25/under= standing-cpu-steal-time-when-should-you-be-worried
>>>>>
>>>>> In short, kill the instance an= d launch a new one. Depending on your latency requirements and operational = ability to respond, you may want to consider paying for dedicated instances= .
>>>>>
>>>>> On Wed, Feb 19, 2014 at 2:30 A= M, Sourabh Agrawal <iitr.soura= bh@gmail.com> wrote:
>>>>>>
>>>>= >> Hi,
>>>>>> I am running cassandra 2.0.3 cluster on 4 AWS node= s. memory arguments are the following for each node :=A0
>>>>= ;>> -Xms8G -Xmx8G -Xmn800M
>>>>>>
>>>= ;>>> I am experiencing consistent high loads on one of the nodes. = Each node is getting approximately equal number of writes. I tried to have = a look at the logs and seems like CMS GC is running every 1-2 seconds.
>>>>>> Any pointers on how to debug this?
>>>= >>> --
>>>>>> Sourabh Agrawal
>>>= >>> Bangalore
>>>>>> +91 9945657973
>&g= t;>>>
>>>>>
>>>>> --
>>>>> ---= --------------
>>>>> Nate McCall
>>>>> = Austin, TX
>>>>> @zznate
>>>>>
>&= gt;>>> Co-Founder & Sr. Technical Consultant
>>>>> Apache Cassandra Consulting
>>>>> http://www.thelastpickle.com
= >>>>
>>>>
>>>> --
>>>= > Sourabh Agrawal
>>>> Bangalore
>>>> +91 9945657973
>>>>
>>
>> --
>> Sourabh Agrawal
>&g= t; Bangalore
>> +91 9945657973
>

--
Sorry this wa= s sent from mobile. Will do less grammar and spell check than usual.
--047d7b33db86e38fff04f2d6f736--