incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Heap stuck at 98% while restarting node
Date Wed, 31 Jul 2013 09:08:50 GMT
> Config : 3 nodes running 1.1.3 (x2) and one node running 1.1.12. No caches, lots of column
families.  memtable_total_space_in_mb set to 4096MB
Reduce this to 2048 so it will flush to disk more frequently and avoid growing the heap so
much. 

From what you've said I would be looking at the size of the row caches and the bloom filters.


> Cassandra is started with a 32GB heap, yes I know it's too high but it works on the other
2 nodes.
Even hippies would say thats high http://www.youtube.com/watch?v=yoSwOrytf_M

Keep an eye on how much time is spent in GC. And remember that the memory you do not allocate
to the JVM is memory that can be used to cache the files. 

Hope that helps. 

-----------------
Aaron Morton
Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 29/07/2013, at 2:25 AM, Philippe <watcherfr@gmail.com> wrote:

> While restarting a 1.1.12 node, I've run into this while it's replaying the commit log:
> GCInspector.java (line 145) Heap is 0.9778097528951517 full.  You may need to reduce
memtable and/or cache sizes.  Cassandra will now flush up to the two largest memtables to
free up memory.  Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't
want Cassandra to do this automatically
> I see it flush one column family but that's not enough as the CPU stays locked at one
thread running at 100%
> 
> Config : 3 nodes running 1.1.3 (x2) and one node running 1.1.12. No caches, lots of column
families.  memtable_total_space_in_mb set to 4096MB
> Cassandra is started with a 32GB heap, yes I know it's too high but it works on the other
2 nodes.
> 
> Can someone explain how I can "reduce memtable size" ? Any other ways of getting this
node to restart apart from increasing the HEAP SIZE ?
> 
> Thanks
> 


Mime
View raw message