incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <>
Subject Re: JVM 7, Cass 1.1.1 and G1 garbage collector
Date Tue, 25 Sep 2012 00:02:43 GMT
Haha Ok.
It is not a total waste, but practically your time is better spent in other
places. The problem is just about everything is a moving target, schema,
request rate, hardware. Generally tuning nudges a couple variables in one
direction or the other and you see some decent returns. But each nudge
takes a restart and a warm up period, and with how Cassandra distributes
requests you likely have to flip several nodes or all of them before you
can see the change! By the time you do that its probably a different day or
week. Essentially finding our if one setting is better then the other is
like a 3 day test in production.

Before c* I used to deal with this in tomcat. Once in a while we would get
a dev that read some article about tuning, something about a new jvm, or
collector. With bright eyed enthusiasm they would want to try tuning our
current cluster. They spend a couple days and measure something and say it
was good "lower memory usage". Meanwhile someone else would come to me and
say "higher 95th response time". More short pauses, fewer long pauses,
great taste, less filing.

Most people just want to roflscale their huroku cloud. Tuning stuff is
sysadmin work and the cloud has taught us that the cost of sysadmins are
needless waste of money.

Just kidding !

But I do believe the default cassandra settings are reasonable and
typically I find that most who look at tuning GC usually need more hardware
and actually need to be tuning something somewhere else.

G1 is the perfect example of a time suck. Claims low pause latency for big
heaps, and delivers something regarded by the Cassandra community (and
hbase as well) that works worse then CMS. If you spent 3 hours switching
tuning knobs and analysing, that is 3 hours of your life you will never get

Better to let SUN and other people worry about tuning (at least from where
I sit)

On Saturday, September 15, 2012, Peter Schuller <>
>> Generally tuning the garbage collector is a waste of time.
> Sorry, that's BS. It can be absolutely critical, when done right, and
> only "useless" when done wrong. There's a spectrum in between.
>> Just follow
>> someone else's recommendation and use that.
> No, don't.
> Most recommendations out there are completely useless in the general
> case because someone did some very specific benchmark under very
> specific circumstances and then recommends some particular combination
> of options. In order to understand whether a particular recommendation
> applies to you, you need to know enough about your use-case that I
> suspect you're better of just reading up on the available options and
> figuring things out. Of course, randomly trying various different
> settings to see which seems to work well may be realistic - but you
> loose predictability (in the face of changing patterns of traffic for
> example) if you don't know why it's behaving like it is.
> If you care about GC related behavior you want to understand how the
> application behaves, how the garbage collector behaves, what your
> requirements are, and select settings based on those requirements and
> how the application and GC behavior combine to produce emergent
> behavior. The "best" GC options may vary *wildly* depending on the
> nature of your cluster and your goals. There are also non-GC settings
> (in the specific case of Cassandra) that affect the interaction with
> the garbage collector, like whether you're using row/key caching, or
> things like phi conviction threshold and/or timeouts. It's very hard
> for anyone to give generalized recommendations. If it weren't,
> Cassandra would ship with The One True set of settings that are always
> the best and there would be no discussion.
> It's very unfortunate that the state of GC in the freely available
> JVM:s is at this point given that there exists known and working
> algorithms (and at least one practical implementation) that avoids it,
> mostly. But, it's the situation we're in. The only way around it that
> I know of if you're on Hotspot, is to have the application behave in
> such a way that it avoids the causes of un-predictable behavior w.r.t.
> GC by being careful about it's memory allocation and *retention*
> profile. For the specific case of avoiding *ever* seeing a full gc, it
> gets even more complex.
> --
> / Peter Schuller (@scode,

View raw message