incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <moshe.kr...@barclays.com>
Subject RE: Advice on memory warning
Date Sun, 21 Apr 2013 07:27:29 GMT
My experience (running C* 1.2.2): 

1. I also observe that this occurs during compaction. 
2. I have never yet seen a node recover from this state. Once it starts complaining about
heap, it starts a death spiral, i.e., futile attempts to fix the situation. Eventually the
node starts running GC for so long that it looks down to the other nodes. 
3. Do you observe a lot of pending MemtablePostFlusher tasks in the log? I believe this is
another symptom.

Like you, I would love to get advice on what causes this and how to avoid it.

-----Original Message-----
From: Michael Theroux [mailto:mtheroux2@yahoo.com] 
Sent: Friday, April 19, 2013 6:00 PM
To: user@cassandra.apache.org
Subject: Advice on memory warning

Hello,

We've recently upgraded from m1.large to m1.xlarge instances on AWS to handle additional load,
but to also relieve memory pressure.  It appears to have accomplished both, however, we are
still getting a warning, 0-3 times a day, on our database nodes:

WARN [ScheduledTasks:1] 2013-04-19 14:17:46,532 GCInspector.java (line 145) Heap is 0.7529240824406468
full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to
the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically

This is happening much less frequently than before the upgrade, but after essentially doubling
the amount of available memory, I'm curious on what I can do to determine what is happening
during this time.  

I am collecting all the JMX statistics.  Memtable space is elevated but not extraordinarily
high.  No GC messages are being output to the log.   

These warnings do seem to be occurring doing compactions of column families using LCS with
wide rows, but I'm not sure there is a direct correlation.    

We are running Cassandra 1.1.9, with a maximum heap of 8G.  

Any advice?
Thanks,
-Mike
_______________________________________________

This message may contain information that is confidential or privileged. If you are not an
intended recipient of this message, please delete it and any attachments, and notify the sender
that you have received it in error. Unless specifically stated in the message or otherwise
indicated, you may not duplicate, redistribute or forward this message or any portion thereof,
including any attachments, by any means to any other person, including any retail investor
or customer. This message is not a recommendation, advice, offer or solicitation, to buy/sell
any product or service, and is not an official confirmation of any transaction. Any opinions
presented are solely those of the author and do not necessarily represent those of Barclays.

This message is subject to terms available at: www.barclays.com/emaildisclaimer and, if received
from Barclays' Sales or Trading desk, the terms available at: www.barclays.com/salesandtradingdisclaimer/.
By messaging with Barclays you consent to the foregoing. Barclays Bank PLC is a company registered
in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.
This email may relate to or be sent from other members of the Barclays group.

_______________________________________________

Mime
View raw message