lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Harrison-Fuller <aharrison-ful...@mintel.com>
Subject Solr OOM Crashes / JVM tuning advice
Date Wed, 11 Apr 2018 10:01:18 GMT
Hey all,

I was wondering if I could get some JVM/GC tuning advice to resolve an
issue that we are experiencing.

Full disclaimer, I am in no way a JVM/Solr expert so any advice you can
render would be greatly appreciated.

Our Solr cloud nodes are having issues throwing OOM exceptions under load.
This issue has only started manifesting itself over the last few months
during which time the only change I can discern is an increase in index
size.  They are running Solr 5.5.2 on OpenJDK version "1.8.0_101".  The
index is currently 58G and the server has 46G of physical RAM and runs
nothing other than the Solr node.

The JVM is invoked with the following JVM options:
-XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
-XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark
-XX:ConcGCThreads=4 -XX:InitialHeapSize=12884901888 -XX:+ManagementServer
-XX:MaxHeapSize=12884901888 -XX:MaxTenuringThreshold=8
-XX:NewRatio=3 -XX:OldPLABSize=16
-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 30000 /data/gnpd/solr/logs
-XX:ParallelGCThreads=4
-XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864
-XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90
-XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers
-XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

These values were decided upon serveral years by a colleague based upon
some suggestions from this mailing group with an index size ~25G.

I have imported the GC logs into GCViewer and attached a link to a
screenshot showing the lead up to a OOM crash.  Interestingly the young
generation space is almost empty before the repeated GC's and subsequent
crash.
https://imgur.com/a/Wtlez

I was considering slowly increasing the amount of heap available to the JVM
slowly until the crashes, any other suggestions?  I'm looking at trying to
get the nodes stable without having issues with the GC taking forever to
run.

Additional information can be provided on request.

Cheers!
Adam

-- 

Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in 
England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for 
our other offices can be found at http://www.mintel.com/office-locations 
<http://www.mintel.com/office-locations>.

This email and any attachments 
may include content that is confidential, privileged 
or otherwise 
protected under applicable law. Unauthorised disclosure, copying, 
distribution 
or use of the contents is prohibited and may be unlawful. If 
you have received this email in error,
including without appropriate 
authorisation, then please reply to the sender about the error 
and delete 
this email and any attachments.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message