incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "B. Todd Burruss" <bburr...@real.com>
Subject Re: cassandra not responding
Date Tue, 16 Mar 2010 20:17:58 GMT
i think i better make sure i understand how the row/key cache works.  i 
currently have both set to 10%.  so if cassandra needs to read data from 
an sstable that has 100 million rows, it will cache 10,000,000 rows of 
data from that sstable?  so if my row is ~4k, then we're looking at 
~40gb used by cache?

Todd Burruss wrote:
> the row/key cache is set to 10%, but memory usage stays good until an 
> anticompaction, hinted handoff, etc starts.  (of course maybe i simply 
> don't pay attention to memory until something bad happens)
>
> doesn't sound like anyone else is having trouble, so i'll keep review my 
> settings for cache, keep testing and try to get something more concrete.
>
> Jonathan Ellis wrote:
>   
>> it's almost certainly GC storming due to memory pressure.  (matching
>> the thread dump / the threads using the CPU in top will confirm this.)
>> reducing your cache sizes might be the best option since you already
>> have a 44GB heap.
>>
>> On Tue, Mar 16, 2010 at 12:17 PM, B. Todd Burruss <bburruss@real.com> wrote:
>>   
>>     
>>> thx, i'll try that next time, already restarted node .. but i will say the
>>> exact thing happened on another node as well.
>>>
>>> Jonathan Ellis wrote:
>>>     
>>>       
>>>> You can still get a thread list w/ jstack, though.
>>>>
>>>> On Tue, Mar 16, 2010 at 11:46 AM, Gary Dusbabek <gdusbabek@gmail.com>
>>>> wrote:
>>>>
>>>>       
>>>>         
>>>>> On Tue, Mar 16, 2010 at 11:39, B. Todd Burruss <bburruss@real.com>
wrote:
>>>>>
>>>>>         
>>>>>           
>>>>>> any other ideas on how to troubleshoot?  i have tried kill -3 <java_pid>
>>>>>> in
>>>>>> the past but don't know where cassandra writes the console out. 
i'll
>>>>>> look
>>>>>> at scripts.
>>>>>>
>>>>>>
>>>>>>           
>>>>>>             
>>>>> I have a sneaking suspicion that unless you're running with '-f' that
>>>>> the thread dump goes into the ether.  The only logical place to look
>>>>> would be the log, which you've probably already done.
>>>>>
>>>>> Gary.
>>>>>
>>>>>
>>>>>         
>>>>>           

Mime
View raw message