incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From srmore <>
Subject Re: heap issues - looking for advices on gc tuning
Date Wed, 30 Oct 2013 13:21:57 GMT
We ran into similar heap issues a while ago for 1.0.11, I am not sure
whether you are at the luxury of upgrading to at-least 1.2.9, we were not.
After a lot of various painful attempts and weeks of testing (just as in
your case) the following settings worked (did not completely relieve the
heap pressure but helped a lot). We still see some heap issues but at-least
it is a bit stable. Unlike in your case we had very heavy reads and writes.
But its good to know that this happens for light load, I was thinking this
was a symptom of heavy load.


Not sure whether this will help you or not but I think its worth a try.


On Wed, Oct 30, 2013 at 4:34 AM, Jason Tang <> wrote:

> What's configuration of following parameters
> memtable_flush_queue_size:
> concurrent_compactors:
> 2013/10/30 Piavlo <>
>> Hi,
>> Below I try to give a full picture to the problem I'm facing.
>> This is a 12 node cluster, running on ec2 with m2.xlarge instances (17G
>> ram , 2 cpus).
>> Cassandra version is 1.0.8
>> Cluster normally having between 3000 - 1500 reads per second (depends on
>> time of the day) and 1700 - 800 writes per second- according to Opscetner.
>> RF=3, now row caches are used.
>> Memory relevant  configs from cassandra.yaml:
>> flush_largest_memtables_at: 0.85
>> reduce_cache_sizes_at: 0.90
>> reduce_cache_capacity_to: 0.75
>> commitlog_total_space_in_mb: 4096
>> relevant JVM options used are:
>> -Xms8000M -Xmx8000M -Xmn400M
>> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>> -XX:MaxTenuringThreshold=1
>> -XX:**CMSInitiatingOccupancyFraction**=80 -XX:+**
>> UseCMSInitiatingOccupancyOnly"
>> Now what happens is that with these settings after cassandra process
>> restart, the GC it working fine at the beginning, and heap used looks like a
>> saw with perfect teeth, eventually the teeth size start to diminish until
>> the teeth become not noticable, and then cassandra starts to spend lot's of
>> CPU time
>> doing gc. It takes about 2 weeks until for such cycle , and then I need
>> to restart cassandra process to improve performance.
>> During all this time there are no memory  related messages in cassandra
>> system.log, except a "GC for ParNew: little above 200ms" once in a while.
>> Things i've already done trying to reduce this eventual heap pressure.
>> 1) reducing bloom_filter_fp_chance  resulting in reduction from ~700MB to
>> ~280MB total per node based on all Filter.db files on the node.
>> 2) reducing key cache sizes, and dropping key_caches for CFs which do no
>> not have many reads
>> 3) the heap size was increased from 7000M to 8000M
>> All these have not really helped , just the increase from 7000M to 8000M,
>> helped in increase the cycle till excessive gc from ~9 days to ~14 days.
>> I've tried to graph overtime the data that is supposed to be in heap vs
>> actual heap size, by summing up all CFs bloom filter sizes + all CFs key
>> cache capacities multipled by average key size + all CFs memtables data
>> size reported (i've overestimated the data size a bit on purpose to be on
>> the safe size).
>> Here is a link to graph showing last 2 day metrics for a node which could
>> not effectively do GC, and then cassandra process was restarted.
>> You can clearly see that before and after restart, the size of data that
>> is in supposed to be in heap, is the same pretty much the same,
>> which makes me think that I really need is GC tunning.
>> Also I suppose that this is not due to number of total keys each node has
>> , which is between 300 - 200 milions keys for all CF key estimates summed
>> on a code.
>> The nodes have datasize between 75G to 45G  accordingly to milions of
>> keys. And all nodes are starting to have having GC heavy load after about
>> 14 days.
>> Also the excessive GC and heap usage are not affected by load which
>> varies depending on time of the day (see read/write rates at the beginning
>> of the mail).
>> So again based on this , I assume this is not due to large number of keys
>> or too much load on the cluster,  but due to a pure GC misconfiguration
>> issue.
>> Things I remember that I've tried for GC tunning:
>> 1) Changing -XX:MaxTenuringThreshold=1 to values like 8 - did not help.
>> 2) Adding  -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing -XX:**
>> CMSIncrementalDutyCycleMin=0
>>                   -XX:CMSIncrementalDutyCycle=10 -XX:ParallelGCThreads=2
>> JVM_OPTS -XX:ParallelCMSThreads=1
>>     this actually made things worse.
>> 3) Adding -XX:-XX-UseAdaptiveSizePolicy -XX:SurvivorRatio=8 - did not
>> help.
>> Also since it takes like 2 weeks to verify that changing GC setting did
>> not help, the process is painfully slow to try all the possibilities :)
>> I'd highly appreciate any help and hints on the GC tunning.
>> tnx
>> Alex

View raw message