incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <>
Subject Re: Questions around the heap
Date Wed, 07 Nov 2012 15:45:06 GMT
I have to say that I have no idea on how to tune them.

I discover the existence of bloom filters a few month ago and even after
reading and I
am not sure what would be the impacts (positives and negatives) of tuning
the bloom filters.

>From my reads I understand that with a bloom_filter_fp_chance > 0 I
introduce a chance to get a false positive from a SSTable inducing
eventually more latency while answering queries but using less memory. Is
that right ?

"What are your bloom filter settings on your CFs?"

They are default (0 - which seems to mean fully enabled

Cant they grow indefinitely or is there a threshold?

Is there a way to "explore" the heap to be sure that bloom filters are
causing this intensive use of the memory inside the heap before tuning them?


"For example, to run an analytics application that heavily scans a
particular column family, you would want to inhibit or disable the Bloom
filter on the column family by setting it high"

Why would I do that, won't it slow the display of analytics?


2012/11/7 Bryan <>

> What are your bloom filter settings on your CFs? Maybe look here:
> On Nov 7, 2012, at 4:56 AM, Alain RODRIGUEZ wrote:
> Hi,
> We just had some issue in production that we finally solve upgrading
> hardware and increasing the heap.
> Now we have 3 xLarge servers from AWS (15G RAM, 4 cpu - 8 cores). We add
> them and then removed the old ones.
> With full default configuration, 0.75 threshold of 4G was being reach
> continuously, so I was obliged to increase the heap to 8G:
> Memtable  : 2G (Manually configured)
> Key cache : 0.1G (min(5% of Heap (in MB), 100MB))
> System     : 1G     (more or less, from datastax doc)
> It should use about 3 G and it actually use between 4 and 6 G.
> So here are my questions:
> How can we know how the heap is being used, monitor it ?
> Why have I that much memory used in the heap of my new servers ?
> All configurations not specified are default from 1.1.2 Cassandra.
> Here is what happen to us before, why we change our hardware, if you have
> any clue on what happen we would be glad to learn and maybe come back to
> our old hardware.
> -------------------------------- User experience
> ------------------------------------------------------------------------
> We had a Cassandra 1.1.2 2 nodes cluster with RF2 and CL.ONE (R&W) running
> on 2 m1.Large aws (7.5G RAM, 2 cpu - 4 cores dedicated to Cassandra only).
> Cassandra.yaml was configured with 1.1.2 default options and in
> I configured a 4G heap with a 200M "new size".
> That is the heap that was supposed to be used.
> Memtable  : 1.4G (1/3 of the heap)
> Key cache : 0.1G (min(5% of Heap (in MB), 100MB))
> System     : 1G     (more or less, from datastax doc)
> So we are around 2.5G max in theory out of 3G usable (threshold 0.75 of
> the heap before flushing memtable because of pressure)
> I thought it was ok regarding Datastax documentation:
> "Regardless of how much RAM your hardware has, you should keep the JVM
> heap size constrained by the following formula and allow the operating
> system’s file cache to do the rest:
> (memtable_total_space_in_mb) + 1GB + (cache_size_estimate)"
> After adding a third node and changing the RF from 2 to 3 (to allow using
> CL.QUORUM and still be able to restart a node whenever we want), things
> went really bad. Even if I still don't get how any of these operations
> could possibly affect the heap needed.
> All the 3 nodes reached the 0.75 heap threshold (I tried to increase it to
> 0.85, but it was still reached). And they never came down. So my cluster
> started flushing a lot and the load increased because of
> unceasing compactions. This unexpected load produced latency that broke
> down our service for a while. Even with the service down, Cassandra was
> unable to recover.

View raw message