Regarding memory usage after a repair ... Are the merkle trees kept around?
On Mon, Oct 22, 2012 at 6:05 PM, aaron morton <email@example.com> wrote:If you restart a node is the onset of GC activity correlated to some event?The GC was on-going even when the nodes were not compacting or running a heavy application load -- even when the main app was paused constant the GC continued.Yes and no. When the nodes were generally under the .75 occupancy threshold a weekly "repair -pr" job would cause them to go over the threshold and then stay there even after the repair had completed and there were no ongoing compactions. It acts as though at least some substantial amount of memory used during repair was never dereferenced once the repair was complete.Once one CF in particular grew larger the constant GC would start up pretty soon (less than 90 minutes) after a node restart even without a repair.How many keys did the CF have per node?As a test we dropped the largest CF and the memory usage immediately dropped to acceptable levels and the constant GC stopped. So it's definitely related to data load. memtable size is 1 GB, row cache is disabled and key cache is small (default).I dismissed the memory used to hold bloom filters and index sampling. That memory is not considered part of the memtable size, and will end up in the tenured heap. It is generally only a problem with very large key counts per node.I've changed the app to retain less data for that CF but I think that it was about 400M rows per node. Row keys are a TimeUUID. All of the rows are write-once, never updated, and rarely read. There are no secondary indexes for this particular CF.The default should be 0.000744.They were 2+ GB (as reported by nodetool cfstats anyway). It looks like the default bloom_filter_fp_chance defaults to 0.0If the chance is zero or null this code should run when a new SSTable is written// paranoia -- we've had bugs in the thrift <-> avro <-> CfDef dance before, let's not let that break thingslogger.error("Bloom filter FP chance of zero isn't supposed to happen");Were the CF's migrated from an old version ?Yes, the CF were created in 1.0.9, then migrated to 1.0.11 and finally to 1.1.5 with a "upgradesstables" run at each upgrade along the way.I could not find a way to view the current bloom_filter_fp_chance settings when they are at a default value. JMX reports the actual fp rate and if a specific rate is set for a CF that shows up in "describe table" but I couldn't find out how to tell what the default was. I didn't inspect the source.Is there any way to predict how much memory the bloom filters will consume if the size of the row keys, number or rows is known, and fp chance is known?See o.a.c.utils.BloomFilter.getFilter() in the codeThis http://hur.st/bloomfilter appears to give similar results.
Ahh, very helpful. This indicates that 714MB would be used for the bloom filter for that one CF.JMX / cfstats reports "Bloom Filter Space Used" but the MBean method name (getBloomFilterDiskSpaceUsed) indicates this is the on-disk space. If on-disk and in-memory space used is similar then summing up all the "Bloom Filter Space Used" says they're currently consuming 1-2 GB of the heap which is substantial.If a CF is rarely read is it safe to set bloom_filter_fp_chance to 1.0? It just means more trips to SSTable indexes for a read correct? Trade RAM for time (disk I/O).-Bryan