incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: 0.7 memory usage problem
Date Wed, 22 Sep 2010 01:56:29 GMT
Disabling swap entirely is usually the easiest fix, yes.

On Mon, Sep 20, 2010 at 8:10 PM, Alaa Zubaidi <alaa.zubaidi@pdf.com> wrote:
>  Thanks Peter,
> I decreased the heap size, it did not help, however, it delayed the problem.
> I noticed that its swapping, so, do you think that I should set windows to
> Not to swap?
>
> Do you think its related to this issue?
> https://issues.apache.org/jira/browse/CASSANDRA-1014
>
> Thanks,
> Alaa
>
> On 9/18/2010 7:04 AM, Peter Schuller wrote:
>>>
>>> I see a spike in heap memory usage on Node 2 where it goes from around 1G
>>> to
>>> 6GB (max) in less than an hour, and then goes our of memory.
>>> There are some errors in the log file that are reported by other people,
>>> but
>>> I don't think that these errors are the reason, because it use to happen
>>> even before I have seen them.
>>>
>>> Can someone help me understand whats going on??
>>
>> Only partially:
>>
>> 54 seconds ParNew GC:s are wild and crazy:
>>
>>    INFO [GC inspection] 2010-09-17 14:53:59,403 GCInspector.java (line
>> 129) GC for ParNew: 54095 ms, 53297952 reclaimed leaving 4712568360
>> used; max is 6563430400
>>
>> Is the machine swapping?
>>
>> I noticed there is hinted hand-off activity going on. Maybe that is a
>> result of nodes dropping in and out due to swapping. In any case, you
>> definitely don't want to have the machine swapping to death. I'm not
>> sure what the best way is to avoid this on Windows, other than
>> decreasing heap size.
>>
>> The repeated exceptions in your log aren't normal as far as I know. I
>> don't remember, but IIRC the UTF-8 encoding issues can be caused by
>> changing partitioner after inserting data (but I'm not sure).
>>
>> With respect to memory use, you don't seem to be inserting so much
>> data for bloom filters and sstable index samples to be a problem.
>> Memtable flushing could cause problems if they're happening too slowly
>> (maybe plausible with swapping) - except that the stage statistics
>> don't indicate there are lots of memtables in memory waiting to be
>> flushed, so that shouldn't be it.
>>
>> Hinted handoff maybe, but I don't remember whether hinted handoff has
>> the potential to accumulate data in RAM (no time to check now).
>>
>> Regardless, I'd recommend fixing any swapping issues you have before
>> trying to draw conclusions about performance. And you don't want those
>> exceptions in your logs.
>>
>
> --
> Alaa Zubaidi
> PDF Solutions, Inc.
> 333 West San Carlos Street, Suite 700
> San Jose, CA 95110  USA
> Tel: 408-283-5639 (or 408-280-7900 x5639)
> fax: 408-938-6479
> email: alaa.zubaidi@pdf.com
>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message