lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Vasilyev <nick.vasily...@gmail.com>
Subject Re: Solr 5.2.1 on Java 8 GC
Date Fri, 29 Apr 2016 14:40:52 GMT
Not sure if it helps anyone, but I am seeing decent results with the
following.

It was mostly a result of trial and error, I am not familiar with Java GC
or even Java itself. I added my interpretation of what was happening, but I
am not sure if it is right, take it for what it's worth. It'd be nice if
someone could provide a better technical explanation. We are about to hit
deaily peak load and so far it doesn't look like there is any negative
performance impact.

-XX:NewRatio=2 \ #Increases the size of the young generation
-XX:SurvivorRatio=3 \ #Increases the size of the survivor spaces
-XX:TargetSurvivorRatio=90 \
-XX:MaxTenuringThreshold=8 \
-XX:+UseConcMarkSweepGC \
-XX:+CMSScavengeBeforeRemark \
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \ #Increasing these didn't
really help
-XX:PretenureSizeThreshold=512m \ # I am not sure what the full impact of
this yet, but I am assuming it will put less stuff in the eden space
-XX:CMSFullGCsBeforeCompaction=1 \
-XX:+UseCMSInitiatingOccupancyOnly \
-XX:CMSInitiatingOccupancyFraction=70 \
-XX:CMSMaxAbortablePrecleanTime=6000 \
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:+UseLargePages \
-XX:+AggressiveOpts \

Here are the GC times over 1 second.

Before:
2016-04-26T04:30:53.175-0400: 244734.856: Total time for which application
threads were stopped: 12.6587130 seconds, Stopping threads took: 0.0024770
seconds
2016-04-26T04:31:22.808-0400: 244764.489: Total time for which application
threads were stopped: 10.6840330 seconds, Stopping threads took: 0.0004840
seconds
2016-04-26T04:31:48.586-0400: 244790.267: Total time for which application
threads were stopped: 10.8198760 seconds, Stopping threads took: 0.0010340
seconds
2016-04-26T04:32:10.095-0400: 244811.777: Total time for which application
threads were stopped: 9.5644690 seconds, Stopping threads took: 0.0006750
seconds
2016-04-26T04:32:32.600-0400: 244834.282: Total time for which application
threads were stopped: 10.0890420 seconds, Stopping threads took: 0.0009930
seconds
2016-04-26T04:32:55.747-0400: 244857.429: Total time for which application
threads were stopped: 10.3426480 seconds, Stopping threads took: 0.0008190
seconds
2016-04-26T04:33:20.522-0400: 244882.203: Total time for which application
threads were stopped: 10.7531070 seconds, Stopping threads took: 0.0013280
seconds
2016-04-26T04:33:45.853-0400: 244907.535: Total time for which application
threads were stopped: 10.3933700 seconds, Stopping threads took: 0.0013970
seconds
2016-04-26T04:34:15.634-0400: 244937.316: Total time for which application
threads were stopped: 10.5744420 seconds, Stopping threads took: 0.0008980
seconds
2016-04-26T04:34:53.802-0400: 244975.484: Total time for which application
threads were stopped: 10.4964470 seconds, Stopping threads took: 0.0013830
seconds
2016-04-26T04:35:19.276-0400: 245000.957: Total time for which application
threads were stopped: 9.8195470 seconds, Stopping threads took: 0.0016110
seconds
2016-04-26T04:35:43.617-0400: 245025.299: Total time for which application
threads were stopped: 9.4856600 seconds, Stopping threads took: 0.0014980
seconds
2016-04-26T04:36:06.540-0400: 245048.222: Total time for which application
threads were stopped: 9.5009880 seconds, Stopping threads took: 0.0009080
seconds
2016-04-26T04:36:32.843-0400: 245074.525: Total time for which application
threads were stopped: 9.6370000 seconds, Stopping threads took: 0.0011770
seconds
2016-04-26T04:36:57.114-0400: 245098.795: Total time for which application
threads were stopped: 10.0064990 seconds, Stopping threads took: 0.0011480
seconds
2016-04-26T04:37:21.074-0400: 245122.755: Total time for which application
threads were stopped: 9.7061140 seconds, Stopping threads took: 0.0009760
seconds
2016-04-26T04:37:45.716-0400: 245147.398: Total time for which application
threads were stopped: 9.9133330 seconds, Stopping threads took: 0.0008220
seconds
2016-04-26T04:38:11.412-0400: 245173.094: Total time for which application
threads were stopped: 10.6839560 seconds, Stopping threads took: 0.0015370
seconds
2016-04-26T04:38:37.177-0400: 245198.859: Total time for which application
threads were stopped: 10.0646910 seconds, Stopping threads took: 0.0013740
seconds
2016-04-26T04:39:00.516-0400: 245222.197: Total time for which application
threads were stopped: 9.8280250 seconds, Stopping threads took: 0.0008900
seconds
2016-04-26T04:39:25.255-0400: 245246.937: Total time for which application
threads were stopped: 10.8429080 seconds, Stopping threads took: 0.0007120
seconds
2016-04-26T04:41:06.937-0400: 245348.619: Total time for which application
threads were stopped: 9.8060420 seconds, Stopping threads took: 0.0006300
seconds
2016-04-26T04:41:43.370-0400: 245385.052: Total time for which application
threads were stopped: 10.8144800 seconds, Stopping threads took: 0.0002260
seconds
2016-04-26T04:42:09.479-0400: 245411.161: Total time for which application
threads were stopped: 9.4059640 seconds, Stopping threads took: 0.0001340
seconds
2016-04-26T04:42:36.033-0400: 245437.715: Total time for which application
threads were stopped: 9.9446430 seconds, Stopping threads took: 0.0007500
seconds
2016-04-26T04:43:02.409-0400: 245464.091: Total time for which application
threads were stopped: 10.4197000 seconds, Stopping threads took: 0.0000260
seconds
2016-04-26T04:43:29.559-0400: 245491.241: Total time for which application
threads were stopped: 9.6712880 seconds, Stopping threads took: 0.0001080
seconds
2016-04-26T04:43:56.648-0400: 245518.330: Total time for which application
threads were stopped: 9.8339590 seconds, Stopping threads took: 0.0011820
seconds
2016-04-26T04:45:35.358-0400: 245617.040: Total time for which application
threads were stopped: 9.5853210 seconds, Stopping threads took: 0.0001760
seconds
2016-04-26T04:54:58.764-0400: 246180.446: Total time for which application
threads were stopped: 2.9048350 seconds, Stopping threads took: 0.0008180
seconds
2016-04-26T04:55:06.107-0400: 246187.789: Total time for which application
threads were stopped: 1.1189760 seconds, Stopping threads took: 0.0011390
seconds

After:
2016-04-29T04:30:05.758-0400: 29962.077: Total time for which application
threads were stopped: 1.0823960 seconds, Stopping threads took: 0.0005840
seconds
2016-04-29T04:30:11.349-0400: 29967.668: Total time for which application
threads were stopped: 1.4147830 seconds, Stopping threads took: 0.0008980
seconds
2016-04-29T04:30:17.198-0400: 29973.517: Total time for which application
threads were stopped: 1.6294590 seconds, Stopping threads took: 0.0009380
seconds
2016-04-29T04:30:22.350-0400: 29978.669: Total time for which application
threads were stopped: 1.6787880 seconds, Stopping threads took: 0.0012320
seconds
2016-04-29T04:30:28.230-0400: 29984.549: Total time for which application
threads were stopped: 1.6895760 seconds, Stopping threads took: 0.0010270
seconds
2016-04-29T04:30:29.944-0400: 29986.263: Total time for which application
threads were stopped: 1.5271500 seconds, Stopping threads took: 0.0009670
seconds
2016-04-29T04:30:35.282-0400: 29991.601: Total time for which application
threads were stopped: 1.6575670 seconds, Stopping threads took: 0.0006200
seconds
2016-04-29T04:30:51.011-0400: 30007.329: Total time for which application
threads were stopped: 2.0383550 seconds, Stopping threads took: 0.0004640
seconds
2016-04-29T04:31:03.032-0400: 30019.351: Total time for which application
threads were stopped: 2.1963570 seconds, Stopping threads took: 0.0004650
seconds
2016-04-29T04:31:07.679-0400: 30023.998: Total time for which application
threads were stopped: 1.2220760 seconds, Stopping threads took: 0.0004720
seconds

On Thu, Apr 28, 2016 at 1:02 PM, Jeff Wartes <jwartes@whitepages.com> wrote:

>
> Shawn Heisey’s page is the usual reference guide for GC settings:
> https://wiki.apache.org/solr/ShawnHeisey
> Most of the learnings from that are in the Solr 5.x startup scripts
> already, but your heap is bigger, so your mileage may vary.
>
> Some tools I’ve used while doing GC tuning:
>
> * VisualVM - Comes with the jdk. It has a Visual GC plug-in that’s pretty
> nice for visualizing what’s going on in realtime, but you need to connect
> it via jstatd for that to work.
> * GCViewer - Visualizes a GC log. The UI leaves a lot to be desired, but
> it’s the best tool I’ve found for this purpose. Use this fork for jdk 6+ -
> https://github.com/chewiebug/GCViewer
> * Swiss Java Knife has a bunch of useful features -
> https://github.com/aragozin/jvm-tools
> * YourKit - I’ve been using this lately to analyze where garbage comes
> from. It’s not free though.
> * Eclipse Memory Analyzer - I used this to analyze heap dumps before I got
> a YourKit license: http://www.eclipse.org/mat/
>
> Good luck!
>
>
>
>
>
>
> On 4/28/16, 9:27 AM, "Yonik Seeley" <yseeley@gmail.com> wrote:
>
> >On Thu, Apr 28, 2016 at 12:21 PM, Nick Vasilyev
> ><nick.vasilyev1@gmail.com> wrote:
> >> Hi Yonik,
> >>
> >> There are a lot of logistics involved with re-indexing and naturally
> >> upgrading Solr. I was hoping that there is an easier alternative since
> this
> >> is only a single back end script that is having problems.
> >>
> >> Is there any room for improvement with tweaking GC params?
> >
> >There always is ;-)  But I'm not a GC tuning expert.  I prefer to
> >attack memory problems more head-on (i.e. with code to use less
> >memory).
> >
> >-Yonik
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message