incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: cassandra not responding
Date Tue, 16 Mar 2010 20:27:07 GMT
Right.

As a rule of thumb you should only use one of {key cache, row cache}
at a time on a given CF.

On Tue, Mar 16, 2010 at 3:17 PM, B. Todd Burruss <bburruss@real.com> wrote:
> i think i better make sure i understand how the row/key cache works.  i
> currently have both set to 10%.  so if cassandra needs to read data from an
> sstable that has 100 million rows, it will cache 10,000,000 rows of data
> from that sstable?  so if my row is ~4k, then we're looking at ~40gb used by
> cache?
>
> Todd Burruss wrote:
>>
>> the row/key cache is set to 10%, but memory usage stays good until an
>> anticompaction, hinted handoff, etc starts.  (of course maybe i simply don't
>> pay attention to memory until something bad happens)
>>
>> doesn't sound like anyone else is having trouble, so i'll keep review my
>> settings for cache, keep testing and try to get something more concrete.
>>
>> Jonathan Ellis wrote:
>>
>>>
>>> it's almost certainly GC storming due to memory pressure.  (matching
>>> the thread dump / the threads using the CPU in top will confirm this.)
>>> reducing your cache sizes might be the best option since you already
>>> have a 44GB heap.
>>>
>>> On Tue, Mar 16, 2010 at 12:17 PM, B. Todd Burruss <bburruss@real.com>
>>> wrote:
>>>
>>>>
>>>> thx, i'll try that next time, already restarted node .. but i will say
>>>> the
>>>> exact thing happened on another node as well.
>>>>
>>>> Jonathan Ellis wrote:
>>>>
>>>>>
>>>>> You can still get a thread list w/ jstack, though.
>>>>>
>>>>> On Tue, Mar 16, 2010 at 11:46 AM, Gary Dusbabek <gdusbabek@gmail.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> On Tue, Mar 16, 2010 at 11:39, B. Todd Burruss <bburruss@real.com>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> any other ideas on how to troubleshoot?  i have tried kill -3
>>>>>>> <java_pid>
>>>>>>> in
>>>>>>> the past but don't know where cassandra writes the console out.
 i'll
>>>>>>> look
>>>>>>> at scripts.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> I have a sneaking suspicion that unless you're running with '-f'
that
>>>>>> the thread dump goes into the ether.  The only logical place to
look
>>>>>> would be the log, which you've probably already done.
>>>>>>
>>>>>> Gary.
>>>>>>
>>>>>>
>>>>>>
>

Mime
View raw message