lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kelly, Frank" <frank.ke...@here.com>
Subject Lucene Merge Thread: skip too large
Date Wed, 18 Jan 2017 13:51:19 GMT
Hello,

We're investigating a strange spike in Heap memory usage in our Production Solr.
Heap is stable for days ~ 1.6GB and then suddenly spikes to 3.9 GB and we get an OOM.

Our app server behavior using Solr appears to unchanged (no new schema updates, no additional
indexing or searching we could see)
We're speculating that perhaps segment merges may be contributing to the heap size increase?

Details
Solr 5.3.1
Solr Cloud deployment with 110M+ documents in 2 Collections (72M and 28M) each across 3 shards
(each with 3 replicas)
Heavy indexing vs Query load (API calls are 90% Indexing, 10% querying)

Heap Settings
-Xmx4096m

Some solrconfig.xml settings

 <!-- default: 100 -->
     <ramBufferSizeMB>256</ramBufferSizeMB>
     <!-- default: 1000 -->
     <maxBufferedDocs>10000</maxBufferedDocs>

     <!-- default: 8 -->
     <maxIndexingThreads>10</maxIndexingThreads>

  <mergeFactor>20</mergeFactor>

We turned on InfoStream logging and saw the following

2017-01-18 13:31:55.368 INFO  (Lucene Merge Thread #24) [c:prod_us-east-1_here_account s:shard1
r:core_node30 x:prod_us-east-1_here_account_shard1_replica4] o.a.s.u.LoggingInfoStream [TMP][Lucene
Merge Thread #24]:   seg=_9eac9(5.3.1):C23776249/1714903:delGen=13735 size=4338.599 MB [skip:
too large]
2017-01-18 13:31:55.368 INFO  (Lucene Merge Thread #24) [c:prod_us-east-1_here_account s:shard1
r:core_node30 x:prod_us-east-1_here_account_shard1_replica4] o.a.s.u.LoggingInfoStream [TMP][Lucene
Merge Thread #24]:   seg=_a9nzh(5.3.1):c1627310/860166:delGen=5448 size=175.146 MB
2017-01-18 13:31:55.368 INFO  (Lucene Merge Thread #24) [c:prod_us-east-1_here_account s:shard1
r:core_node30 x:prod_us-east-1_here_account_shard1_replica4] o.a.s.u.LoggingInfoStream [TMP][Lucene
Merge Thread #24]:   seg=_aplxl(5.3.1):c1091264/339932:delGen=1187 size=172.199 MB
2017-01-18 13:31:55.368 INFO  (Lucene Merge Thread #24) [c:prod_us-east-1_here_account s:shard1
r:core_node30 x:prod_us-east-1_here_account_shard1_replica4] o.a.s.u.LoggingInfoStream [TMP][Lucene
Merge Thread #24]:   seg=_aqmq8(5.3.1):c122591/55247:delGen=990 size=15.848 MB

Is this "skip: too large" indicative of some problem? Is the size of that segment (relative
to our Heap) problematic?

Thanks!

-Frank
[Description: Macintosh HD:Users:jerchow:Downloads:Asset_Package_01_160721:HERE_Logo_2016:sRGB:PDF:HERE_Logo_2016_POS_sRGB.pdf]



Frank Kelly

Principal Software Engineer



HERE

5 Wayside Rd, Burlington, MA 01803, USA

42° 29' 7" N 71° 11' 32" W

[Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_360.gif]<http://360.here.com/>
   [Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Twitter.gif]
<https://www.twitter.com/here>    [Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_FB.gif]
<https://www.facebook.com/here>     [Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_IN.gif]
<https://www.linkedin.com/company/heremaps>     [Description: /Users/nussbaum/_WORK/PROJECTS/20160726_HERE_EMail_Signature/_Layout/_Images/20160726_HERE_EMail_Signature_Insta.gif]
<https://www.instagram.com/here/>

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message