lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emir Arnautovic <emir.arnauto...@sematext.com>
Subject Re: Spike in SOLR Process and Frequent GC
Date Wed, 31 Aug 2016 10:03:27 GMT
Hi Thiru,

Two Solrs with different data and usage patterns should be tuned 
separately and comparing one to another does not give much value.

Like Shawn suggested, first thing that you can try is increase heap 
size. Having different Xms and Xmx is bad practice so make sure it is 
set to the same value. Even you have enough RAM on your machines, start 
with smaller heap (e.g. 4GB) and monitor JVM. Heap should be just about 
large to handle your top load.

You could also use more advanced monitoring tool that will allow you to 
correlate Solr activities with JVM/OS metrics. One such tool is our's 
SPM: http://sematext.com/spm.

Regards,
Emir

-- 
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On 31.08.2016 06:28, Thiru M wrote:
> Thanks for your response *Shawn*.
>
> Have responded your queries and shared the SOLR details here.
>
> What precisely were you measuring during the night with no activity ?
>
> ·         Earlier we have enabled delta content indexing (content which got
> updated on day to day basis) script which triggers every 30 minutes. We
> monitored the load on SOLR when the delta indexing is performed.
>
> ·         CPU utilization by SOLR process,
>
> ·         Heap memory usage.
>
> what precise methods were you using to measure it?
>
> ·         We used the “*top*” command to monitor the solr process,
>
> ·         We monitored the free space in the server during the delta change
> indexing process,
>
> ·         We also enabled JMX remote monitoring in SOLR and used *jConsole*
> & *jVisualVM* to monitor CPU, Heap & Thread utilizations.
>
> What part of the information you obtained represents a problem in your mind?
>
> ·         Information obtained from top and the SOLR GC log in the *linux-a*
> server
>
> ·         Allocated max heap size is 2 GB. Based on the jConsole monitor h*eap
> usage is < 600 MB in linux-a server but the CPU utilization by SOLR process
> is on top (in figures it consumes 0.5 to 5 %)*.
>
>
> There is one solr instance per server i.e in linux-a one solr instance and
> in linux-b one solr instance (stand alone) is available.
>
>
>
> linux-a
>
> linux-b
>
> Number of Cores
>
> 49
>
> 14
>
> Number of Documents
>
> 6145495 (~6M)
>
> 1923181  (~1M)
>
> Overall index directory size
>
> 9.9 GB
>
> 14 GB
>
> Min Heap Memory
>
> 256 MB
>
> Max Heap Memory
>
> 2048 MB
>
> Total # System Processors
>
> 40
>
> Overall RAM size
>
> 125 GB
>
> Java Version
>
> 1.8.0_65 64-Bit Server VM
>
> SOLR Parameters
>
> -Xms512m
> -Xmx2048m
> -XX:NewRatio=3
> -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90
> -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC
> -XX:ConcGCThreads=4
> -XX:ParallelGCThreads=4
> -XX:+CMSScavengeBeforeRemark
> -XX:PretenureSizeThreshold=64m
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=50
> -XX:CMSMaxAbortablePrecleanTime=6000
> -XX:+CMSParallelRemarkEnabled
> -XX:+ParallelRefProcEnabled
> -verbose:gc
> -XX:+PrintHeapAtGC
> -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps
> -XX:+PrintGCTimeStamps
> -XX:+PrintTenuringDistribution
> -XX:+PrintGCApplicationStoppedTime
> -Xloggc:/app/solr/logs/solr_gc.log
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.local.only=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.port=17010
> -Dcom.sun.management.jmxremote.rmi.port=17010
> -Djetty.port=7010
> -DSTOP.PORT=6010
> -DSTOP.KEY=solrrocks
> -Duser.timezone=UTC
> -Djetty.home=/opt/solr/server
> -Dsolr.solr.home=/local/solr/default/data
> -Dsolr.install.dir=/opt/solr
> -Dlog4j.configuration=file:/app/solr/log4j.properties
> -Xss256k
>
>
> Regards,
>
> Thirukumaran M
>
>
> On Sun, Aug 28, 2016 at 2:01 AM, Shawn Heisey <apache@elyograg.org> wrote:
>
>> On 8/27/2016 9:08 AM, Thiru M wrote:
>>> We are using Solr 5.4.0 - "stand-alone" mode in our production boxes
>>> hosted in Red Hat Enterprise Linux (RHEL) OS.
>>>
>>> Each box have number of different cores. Have attached the screenshot
>>> shot with the Solr core & system details.
>>>
>>> 1. Earlier indexing was performed every 30 minutes in both production
>>> servers,
>>>
>>> 2. In linux-a server 30 (stand-alone) cores created on same day and
>>> content indexed into it,
>>>
>>> 3. we then spotted unusual GC performing every 2 to 7 seconds in
>>> linux-a server and the  Solr process spiked,
>>>
>>> 4. Then we removed the indexing in linux-a server for a week,
>>> monitored the both Solr process and GC.(No indexing performed during
>>> this time),
>>>
>>> 5. No one uses the system during night time which we ensured it from
>>> our end. But both Solr process and GC were in its peak, even "during
>>> non business hours",
>>>
>>> 6. Have restated Solr instance in linux-a server. GC started again
>>> after Solr instance brought up,
>>>
>>> 7. In linux-b server no spike in Solr process and no issues with GC,
>>>
>> Indexing creates a LOT of garbage.  Queries also create garbage, but not
>> nearly as fast as indexing.  Solr has some background processes, and
>> these will create garbage too.  Java uses a garbage collection memory
>> model, so this is completely normal for java applications.
>>
>> What precisely were you measuring during the night with no activity, and
>> what precise methods were you using to measure it?  What part of the
>> information you obtained represents a problem in your mind?
>>
>> We'll also need some details about these servers:
>>
>> * Total index size of all Solr cores on the server.
>> * Total amount of memory installed in the server.
>> * Total number of documents contained in all Solr cores.
>> * How many Solr instances per server?  What is the max heap size of each
>> instance?
>>
>> Attachments rarely make it to the list.  The screenshot you mentioned
>> did not make it.  You'll need to put it somewhere on the Internet an
>> provide a URL.  Sharing sites like drobox or imgur are good choices for
>> image data.
>>
>> Since I don't have a clear idea of what the exact issue is here, I don't
>> have any immediate suggestions, aside from possibly increasing your max
>> heap size ... but depending on the answers to the questions above, that
>> might make things worse.
>>
>> Thanks,
>> Shawn
>>
>>
>

Mime
View raw message