lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Walter Underwood <wun...@wunderwood.org>
Subject Re: GC tuning question - can improving GC pauses cause indexing to slow down?
Date Fri, 09 Jan 2015 05:55:30 GMT
I would not be surprised at all. Optimizing for minimum pauses usually increases overhead that
decreases overall throughput. This is a pretty common tradeoff.

For maximum throughput, when you don’t care about pauses, the simplest non-concurrent GC
is often the best. That might be the right choice for running big map-reduce jobs, for example.

Low-pause GCs do lots of extra work in parallel. Some of that work is making guesses which
get thrown away, or doing “just in case” analysis.

To quote Oracle:

"When you evaluate or tune any garbage collection, there is always a latency versus throughput
trade-off. The G1 GC is an incremental garbage collector with uniform pauses, but also more
overhead on the application threads. The throughput goal for the G1 GC is 90 percent application
time and 10 percent garbage collection time. When you compare this to Java HotSpot VM's throughput
collector, the goal there is 99 percent application time and 1 percent garbage collection
time.”

http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html

wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/


On Jan 8, 2015, at 8:53 PM, Shawn Heisey <apache@elyograg.org> wrote:

> Is it possible that tuning garbage collection to achieve much better
> pause characteristics might actually *decrease* index performance?
> 
> Rebuilds that I did while still using a tuned CMS config would take
> between 5.5 and 6 hours, sometimes going slightly over 6 hours.
> 
> A rebuild that I did recently with G1 took 6.82 hours.  A rebuild that I
> did yesterday with further tuned G1 settings (which seemed to result in
> much smaller pauses than the previous G1 settings) took 8.97 hours, and
> that was on slightly faster hardware than the rebuild that took 6.82 hours.
> 
> These rebuilds are done with DIH from MySQL.
> 
> It seems completely counter-intuitive that settings which show better GC
> pause characteristics would result in indexing performance going down
> ... so can anyone shed light on this, tell me whether I'm out of my mind?
> 
> Thanks,
> Shawn
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message