zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Crayford <tcrayf...@heroku.com>
Subject Re: garbage collector choice and tuning
Date Sat, 14 May 2016 13:35:51 GMT
We've been running run hundreds of zookeeper clusters with G1 on relatively
small heaps (some even as low as 2GB) with almost no tuning and had a very
pleasant time so far. Granted, our use is relatively light most of the
time, but still, GC has just not been an issue for us since we switched to
the G1

Tom Crayford
Heroku Kafka

On Fri, May 13, 2016 at 11:15 PM, Patrick Hunt <phunt@apache.org> wrote:

> Please do report your experiences to the list. I think generally folks
> would be interested (I would).
>
> My 0.02 - with such small heap sizes with ZK I typically use CMS. It's
> pretty much fire/forget with the defaults. My experience with G1 (granted
> typically larger heaps) is that it requires more tuning to get the benefit.
> YMMV.
>
> Patrick
>
>
> On Fri, May 13, 2016 at 11:42 AM, Attila Szabo <aszabo@cloudera.com>
> wrote:
>
> > Hi all,
> >
> > I do have one specific story worth mentioning in this subject:
> > About one year ago I was working for a financial company, where we had to
> > maintain a system responsible tons of EOD stock related calculations. The
> > system was working quite okay, but during the quarterly rebalances we
> > always experienced serious performance issues. We were still using CMS
> and
> > just 12GB mem, and regardless the advices provided my team to do GC
> tuning
> > and put more mem to the machine we were not allowed to change anything.
> > Until a part when by the next rebalancing date we hit an 8 minute SLA
> miss
> > (you could imagine what kind of complain storm it has started from our
> > customers...).
> >
> > So I've spent 1 weekend in the office with analyzing GC logs, playing
> with
> > GC params, and learning about G1.
> >
> > The result was the following:
> > After I'd totally understood how our application was working at that
> time I
> > was able to tune it (without touching the code) to work perfectly with
> the
> > 12G scenario without even having any stop the world event and shrank the
> > full runtime below the average runtime (so not the worst case! but
> average
> > everyday runtime) with 2 minutes. The throughput increased from 68% up to
> > 99.8%. I had to be aware that our heap contained a quite big amount of
> old
> > objects needed to be used, thus I had to set bigger the
> > -XX:InitiatingHeapOccupancyPercent
> > and -XX:G1MixedGCLiveThresholdPercent higher, and AFAIR I've also set
> > -XX:MaxGCPauseMillis
> > to 500ms (but in this last I'm not totally sure).
> >
> > When I've switched to Java8 and turned on string deduplication the
> > performance got even better. (With that turned on that specific usecase
> was
> > even okay with just 10GB of memory)
> >
> > The best resources I've used for my journeys were:
> > https://github.com/chewiebug/GCViewer
> > and
> > http://www.infoq.com/news/2015/12/java-garbage-collection-minibook
> (since
> > removed but you could find it here
> https://www.reddit.com/comments/3d8nfo
> > )
> >
> > Since that time I really in love with G1 and I'm very willing to use it
> for
> > small heap size cases (like the one you'd depicted) and nearly a must for
> > big heap size cases.
> >
> > However I did learn a very pragmatic approach during this journey.
> >
> > In all of the cases when you plan a change like this follow this
> approach:
> >
> >    1. Build an easily repeatable testcase for measurements.
> >    2. Measure with both of the old and new GC settings
> >    3. Analyze the logs with GC viewer. Understand how your application
> >    works from consumption POV
> >    4. Repeat step 2 and 3 until you can tune any of the parameters with
> >    each GC algorithm
> >    5. Use the better one.
> >
> > About the massive load of short sessions scenario I would consider
> playing
> > with -XX:G1HeapRegionSize and -XX:G1MaxNewSizePercent params as a start,
> > and check the results.
> >
> > My 2 cents,
> >
> > Cheers,
> > Maugli
> >
> > p.s.: If you're interested I happily advise my aid with the measurements
> > and tuning.
> >
> >
> > On Fri, May 13, 2016 at 4:18 PM, Guy Laden <guy.laden@gmail.com> wrote:
> >
> > > Hi, We are considering CMS vs G1 for ZooKeeper running under Oracle
> JDK8.
> > > The expected heap size is 4-6GB.
> > > How workload-specific is this choice in your opinion and it what ways?
> > E.g.
> > > if many short sessions prefer G1, etc...
> > > Has anybody had experience they're willing to share regarding this?
> > > We'd also be very interested to hear about any gc-tuning flags you've
> had
> > > good experience with.
> > > Thanks
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Attila Szabo
> > Sotware Engineer
> >
> > <http://www.cloudera.com>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message