incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: High CPU load on one node in the cluster
Date Thu, 20 Feb 2014 14:03:22 GMT
Upgrade from 2.0.3. There are several bugs,

On Wednesday, February 19, 2014, Yogi Nerella <ynerella999@gmail.com> wrote:
> You should start your Cassandra daemon with -verbose:gc (please check
syntax) and then run it in foreground, as Cassandra closes the standard out)
> Please see other emails in this forum for getting Garbage Collection
Statistics from Cassandra user mail, or look at any Java specific sites.
> Ex:
http://stackoverflow.com/questions/1161647/how-to-redirect-verbose-garbage-collection-output-to-a-file
>
>
> It depends on what JVM you are running.
>
>
> On Wed, Feb 19, 2014 at 9:12 AM, Sourabh Agrawal <iitr.sourabh@gmail.com>
wrote:
>>
>> How do I get that statistic?
>>
>> On Wed, Feb 19, 2014 at 10:34 PM, Yogi Nerella <ynerella999@gmail.com>
wrote:
>>>
>>> Could be your -Xmn800M is too low, that is why it is trying garbage
collecting very frequently.
>>> Do you have any statistics on how much memory it is collecting on every
cycle?
>>>
>>>
>>> On Wed, Feb 19, 2014 at 8:47 AM, Sourabh Agrawal <iitr.sourabh@gmail.com>
wrote:
>>>>
>>>> Below is CPU usage from top. I don't see any steal. Idle time is
pretty low.
>>>> Cpu(s): 83.3%us, 14.5%sy,  0.0%ni,  0.5%id,  0.0%wa,  0.0%hi,  1.7%si,
 0.0%st
>>>>
>>>> Any other pointers?
>>>>
>>>> On Wed, Feb 19, 2014 at 8:34 PM, Nate McCall <nate@thelastpickle.com>
wrote:
>>>>>
>>>>> You may be seeing steal from another tenant on the VM. This article
has a good explanation:
>>>>>
http://blog.scoutapp.com/articles/2013/07/25/understanding-cpu-steal-time-when-should-you-be-worried
>>>>>
>>>>> In short, kill the instance and launch a new one. Depending on your
latency requirements and operational ability to respond, you may want to
consider paying for dedicated instances.
>>>>>
>>>>> On Wed, Feb 19, 2014 at 2:30 AM, Sourabh Agrawal <
iitr.sourabh@gmail.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> I am running cassandra 2.0.3 cluster on 4 AWS nodes. memory
arguments are the following for each node :
>>>>>> -Xms8G -Xmx8G -Xmn800M
>>>>>>
>>>>>> I am experiencing consistent high loads on one of the nodes. Each
node is getting approximately equal number of writes. I tried to have a
look at the logs and seems like CMS GC is running every 1-2 seconds.
>>>>>> Any pointers on how to debug this?
>>>>>> --
>>>>>> Sourabh Agrawal
>>>>>> Bangalore
>>>>>> +91 9945657973
>>>>>
>>>>>
>>>>> --
>>>>> -----------------
>>>>> Nate McCall
>>>>> Austin, TX
>>>>> @zznate
>>>>>
>>>>> Co-Founder & Sr. Technical Consultant
>>>>> Apache Cassandra Consulting
>>>>> http://www.thelastpickle.com
>>>>
>>>>
>>>> --
>>>> Sourabh Agrawal
>>>> Bangalore
>>>> +91 9945657973
>>
>>
>>
>> --
>> Sourabh Agrawal
>> Bangalore
>> +91 9945657973
>

-- 
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.

Mime
View raw message