incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ralph Goers <ralph.go...@dslextreme.com>
Subject Re: Advice on memory warning
Date Tue, 23 Apr 2013 15:52:30 GMT
We are using DSE, which I believe is also 1.1.9.  We have basically had a non-usable cluster
for months due to this error.  In our case, once it starts doing this it starts flushing sstables
to disk and eventually fills up the disk to the point where it can't compact.  If we catch
it soon enough and restart the node it usually can recover.

In our case, the heap size is 12 GB. As I understand it Cassandra will give 1/3 of that for
sstables. I then noticed that we have one column family that is using nearly 4GB in bloom
filters on each node.  Since the nodes will start doing this when the heap reaches 9GB we
essentially only have 1GB of free memory so when compactions, cleanups, etc take place this
situation starts happening.  We are working to change our data model to try to resolve this.

Ralph 

On Apr 19, 2013, at 8:00 AM, Michael Theroux wrote:

> Hello,
> 
> We've recently upgraded from m1.large to m1.xlarge instances on AWS to handle additional
load, but to also relieve memory pressure.  It appears to have accomplished both, however,
we are still getting a warning, 0-3 times a day, on our database nodes:
> 
> WARN [ScheduledTasks:1] 2013-04-19 14:17:46,532 GCInspector.java (line 145) Heap is 0.7529240824406468
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
> 
> This is happening much less frequently than before the upgrade, but after essentially
doubling the amount of available memory, I'm curious on what I can do to determine what is
happening during this time.  
> 
> I am collecting all the JMX statistics.  Memtable space is elevated but not extraordinarily
high.  No GC messages are being output to the log.   
> 
> These warnings do seem to be occurring doing compactions of column families using LCS
with wide rows, but I'm not sure there is a direct correlation.    
> 
> We are running Cassandra 1.1.9, with a maximum heap of 8G.  
> 
> Any advice?
> Thanks,
> -Mike


Mime
View raw message