incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Trying to write when at cassandra capacity
Date Wed, 10 Jul 2013 23:56:22 GMT
> It hits an OOM.
To add a little more colour because I stepped through this with someone the other day.

When memtables are not removed from the memtable flush queue (because they have not been written)
the queue will fill up. When this happens the flush process will block trying to fill the
queue, and will hold the internal switch lock used to syncronise around the commit log. This
will prevent write threads from progressing. All the while writes will continue to be delivered
to the node and the Mutation thread pool queue will fill. 

All of this results in extreme memory pressure, the JVM will spend a lot of time running GC
to try and free some space. While all the GC is going on chances are the other nodes will
see the failing node ad flapping as it fails to keep up with gossip. None of this will work
and eventually the JVM will raise an OOM error that is normally trapped and results in the
node trying to shut down. During the shutdown process it will try to disable the rpc / native
transports and gossip.

It's a simple thing to test and a useful example to walk through (by looking at the logs)
with an Ops team if they are just starting out. 

Cheers
 
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 10/07/2013, at 5:36 AM, Robert Coli <rcoli@eventbrite.com> wrote:

> On Mon, Jul 8, 2013 at 5:58 PM, Faraaz Sareshwala <fsareshwala@quantcast.com> wrote:
> What does cassandra do when it is at its data capacity (disk drives and memtable
> is full) and writes continue to pour in? My intuition says that cassandra won't
> be able to handle the new writes (they will either get silently dropped or
> cassandra will hit an OOM -- does anyone know which one?). The sstables on disk
> won't magically disappear so cassandra won't be able to service the write
> requests.
> 
> It hits an OOM.
> 
> =Rob
>  


Mime
View raw message