cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "B. Todd Burruss" <bburr...@real.com>
Subject Re: cassandra not responding
Date Tue, 16 Mar 2010 20:04:18 GMT
the row/key cache is set to 10%, but memory usage stays good until an 
anticompaction, hinted handoff, etc starts.  (of course maybe i simply 
don't pay attention to memory until something bad happens)

doesn't sound like anyone else is having trouble, so i'll keep review my 
settings for cache, keep testing and try to get something more concrete.

Jonathan Ellis wrote:
> it's almost certainly GC storming due to memory pressure.  (matching
> the thread dump / the threads using the CPU in top will confirm this.)
> reducing your cache sizes might be the best option since you already
> have a 44GB heap.
>
> On Tue, Mar 16, 2010 at 12:17 PM, B. Todd Burruss <bburruss@real.com> wrote:
>   
>> thx, i'll try that next time, already restarted node .. but i will say the
>> exact thing happened on another node as well.
>>
>> Jonathan Ellis wrote:
>>     
>>> You can still get a thread list w/ jstack, though.
>>>
>>> On Tue, Mar 16, 2010 at 11:46 AM, Gary Dusbabek <gdusbabek@gmail.com>
>>> wrote:
>>>
>>>       
>>>> On Tue, Mar 16, 2010 at 11:39, B. Todd Burruss <bburruss@real.com>
wrote:
>>>>
>>>>         
>>>>> any other ideas on how to troubleshoot?  i have tried kill -3 <java_pid>
>>>>> in
>>>>> the past but don't know where cassandra writes the console out.  i'll
>>>>> look
>>>>> at scripts.
>>>>>
>>>>>
>>>>>           
>>>> I have a sneaking suspicion that unless you're running with '-f' that
>>>> the thread dump goes into the ether.  The only logical place to look
>>>> would be the log, which you've probably already done.
>>>>
>>>> Gary.
>>>>
>>>>
>>>>         

Mime
View raw message