incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: cassandra not responding
Date Tue, 16 Mar 2010 20:22:07 GMT
it could be the "really big row during compaction" limitation, then, too.

On Tue, Mar 16, 2010 at 3:04 PM, B. Todd Burruss <bburruss@real.com> wrote:
> the row/key cache is set to 10%, but memory usage stays good until an
> anticompaction, hinted handoff, etc starts.  (of course maybe i simply don't
> pay attention to memory until something bad happens)
>
> doesn't sound like anyone else is having trouble, so i'll keep review my
> settings for cache, keep testing and try to get something more concrete.
>
> Jonathan Ellis wrote:
>>
>> it's almost certainly GC storming due to memory pressure.  (matching
>> the thread dump / the threads using the CPU in top will confirm this.)
>> reducing your cache sizes might be the best option since you already
>> have a 44GB heap.
>>
>> On Tue, Mar 16, 2010 at 12:17 PM, B. Todd Burruss <bburruss@real.com>
>> wrote:
>>
>>>
>>> thx, i'll try that next time, already restarted node .. but i will say
>>> the
>>> exact thing happened on another node as well.
>>>
>>> Jonathan Ellis wrote:
>>>
>>>>
>>>> You can still get a thread list w/ jstack, though.
>>>>
>>>> On Tue, Mar 16, 2010 at 11:46 AM, Gary Dusbabek <gdusbabek@gmail.com>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> On Tue, Mar 16, 2010 at 11:39, B. Todd Burruss <bburruss@real.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> any other ideas on how to troubleshoot?  i have tried kill -3
>>>>>> <java_pid>
>>>>>> in
>>>>>> the past but don't know where cassandra writes the console out.  i'll
>>>>>> look
>>>>>> at scripts.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> I have a sneaking suspicion that unless you're running with '-f' that
>>>>> the thread dump goes into the ether.  The only logical place to look
>>>>> would be the log, which you've probably already done.
>>>>>
>>>>> Gary.
>>>>>
>>>>>
>>>>>
>

Mime
View raw message