incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David McNelis <dmcne...@gmail.com>
Subject Re: cassandra not responding, log full of gc invocations
Date Wed, 18 Sep 2013 12:30:37 GMT
It is a little more involved than just changing the heap size.  Every
cluster is different, so there isn't much of a set formula.  Some areas to
look into, though:

**Caveat, we're still running in the 1.2 branch and 2.0 has some
differences in what is on versus off heap memory usage, but the basic ideas
should remain the same.**

1) What do you have your memtables set to flush, default, I believe, is 75%
of heap space, lowering this amount is a safety valve, but not a silver
bullet. (From the yaml docs:  Setting this lower than
CMSInitiatingOccupancyFraction is not likely to be useful)
2) Second emergency valve (also from the yaml docs) is the defaults of
    reduce_cache_sizes_at: 0.85
    reduce_cache_capacity_to: 0.6
Similar to flushing memtables, setting it lower than your CMSIOF is likely
not going to give you much gains.

3) Version of java and the garbage collector you are using can make a
pretty big difference.
  a)  Be sure to use Sun's JDK, not OpenJDK
  b)  Some people have had better experience using Java 7 (we are using it
and are pretty happy with the results so far), but it is not officially
supported yet
  c) You can try using GarbageFirst as your GC... We tried this and while
it didn't cause us problems, it didn't end up being the solution
  d) Play with the GC params in cassandra-env.sh on a single node or test
node and see what happens.  For our installation, we've found a sweet spot
of:

VM_OPTS="$JVM_OPTS -XX:+UseParNewGC"
JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC"
JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled"
JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=50"
JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
JVM_OPTS="$JVM_OPTS -XX:+CMSIncrementalMode"
JVM_OPTS="$JVM_OPTS -XX:+CMSIncrementalPacing"
JVM_OPTS="$JVM_OPTS -XX:CMSIncrementalDutyCycleMin=0"
JVM_OPTS="$JVM_OPTS -XX:+CMSIncrementalDutyCycle=10"

with an 8gb heap size on 16gb ram.

One thing to keep in mind before setting a high heap size is that you are,
in many cases, delaying your problem... it will just take longer before you
get stuck in GC hell. The nice thing about smaller heaps is that there
isn't as much GC to be done because there isn't as much ram to be collected.

Last, prior to G1 in Java 7, its been pretty widely documented that GC
performance starts to drop dramatically when the heap is > 8gb.   The
algorithm in G1 is supposed to be much better at handling this than the
other options, so if you've got a shit-ton of ram available, that might be
something worth looking into.


On Wed, Sep 18, 2013 at 8:10 AM, Alexander Shutyaev <shutyaev@gmail.com>wrote:

> Hi all!
>
> We have a problem with cassandra 2.0. We have installed cassandra from
> datastax community respository. We haven't changed any java options from
> the default ones. De-facto Xmx is 1GB. Recently we have encountered a
> couple of cases when cassandra stopped responding and the log was showing
> some constant gc activity. My assumption is that the reason we are having
> this is because of low Xmx. Although we don't have much load on this server
> we do have a lot of column families (cql tables). If I understand correctly
> each column family takes some memory for service purposes. Is that so? Is
> there any Xmx recommendation formula that takes into account the column
> family count and the read/write ops?
>
> Thanks in advance!
>

Mime
View raw message