incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yogi Nerella <ynerella...@gmail.com>
Subject Re: High CPU load on one node in the cluster
Date Wed, 19 Feb 2014 18:02:15 GMT
You should start your Cassandra daemon with -verbose:gc (please check
syntax) and then run it in foreground, as Cassandra closes the standard out)
Please see other emails in this forum for getting Garbage Collection
Statistics from Cassandra user mail, or look at any Java specific sites.

Ex:
http://stackoverflow.com/questions/1161647/how-to-redirect-verbose-garbage-collection-output-to-a-file


It depends on what JVM you are running.



On Wed, Feb 19, 2014 at 9:12 AM, Sourabh Agrawal <iitr.sourabh@gmail.com>wrote:

> How do I get that statistic?
>
>
> On Wed, Feb 19, 2014 at 10:34 PM, Yogi Nerella <ynerella999@gmail.com>wrote:
>
>> Could be your -Xmn800M is too low, that is why it is trying garbage
>> collecting very frequently.
>> Do you have any statistics on how much memory it is collecting on every
>> cycle?
>>
>>
>>
>> On Wed, Feb 19, 2014 at 8:47 AM, Sourabh Agrawal <iitr.sourabh@gmail.com>wrote:
>>
>>> Below is CPU usage from top. I don't see any steal. Idle time is pretty
>>> low.
>>>
>>> Cpu(s): 83.3%us, 14.5%sy,  0.0%ni,  0.5%id,  0.0%wa,  0.0%hi,  1.7%si,
>>>  0.0%st
>>>
>>> Any other pointers?
>>>
>>>
>>> On Wed, Feb 19, 2014 at 8:34 PM, Nate McCall <nate@thelastpickle.com>wrote:
>>>
>>>> You may be seeing steal from another tenant on the VM. This article has
>>>> a good explanation:
>>>>
>>>> http://blog.scoutapp.com/articles/2013/07/25/understanding-cpu-steal-time-when-should-you-be-worried
>>>>
>>>> In short, kill the instance and launch a new one. Depending on your
>>>> latency requirements and operational ability to respond, you may want to
>>>> consider paying for dedicated instances.
>>>>
>>>>
>>>> On Wed, Feb 19, 2014 at 2:30 AM, Sourabh Agrawal <
>>>> iitr.sourabh@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am running cassandra 2.0.3 cluster on 4 AWS nodes. memory arguments
>>>>> are the following for each node :
>>>>> -Xms8G -Xmx8G -Xmn800M
>>>>>
>>>>> I am experiencing consistent high loads on one of the nodes. Each node
>>>>> is getting approximately equal number of writes. I tried to have a look
at
>>>>> the logs and seems like CMS GC is running every 1-2 seconds.
>>>>>
>>>>> Any pointers on how to debug this?
>>>>>
>>>>> --
>>>>> Sourabh Agrawal
>>>>> Bangalore
>>>>> +91 9945657973
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -----------------
>>>> Nate McCall
>>>> Austin, TX
>>>> @zznate
>>>>
>>>> Co-Founder & Sr. Technical Consultant
>>>> Apache Cassandra Consulting
>>>> http://www.thelastpickle.com
>>>>
>>>
>>>
>>>
>>> --
>>> Sourabh Agrawal
>>> Bangalore
>>> +91 9945657973
>>>
>>
>>
>
>
> --
> Sourabh Agrawal
> Bangalore
> +91 9945657973
>

Mime
View raw message