lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Performance warning overlapping onDeckSearchers
Date Wed, 12 Aug 2015 15:18:31 GMT
Something's not adding up here. Is your _client_ perhaps issuing
commits when you index documents? This is Joel's question, so we need
to see how you send docs to Solr. We really need to know how you're
indexing docs to Solr.

My bet (and I suspect Joel's) is that you're either using SolrJ to
send docs to Solr and have something like
****solrJ
while (more docs) {
   create a doc
   send it to Solr
   commit
}

rather than
while (more docs) {
   create a bunch of docs (I usually start with 1,000) using the
"commitwithin" option and make it as long as possible
   send the batch to solr
}

Maybe commit at the very end, but that's not necessary if you're
willing to wait for "commitWithin".

****post.jar
Or, you're using post.jar in some kind of loop which commits every
time you use it by default. You can disable this, try 'java -jar
post.jar -help' for all the options, but the one you want is
-Dcommit=no.

NOTE: you have to issue a commit _sometime_ to see the docs, either
the commitWithin option in SolrJ or explicitly if you're using the
post.jar tool. You can even issue a commit (this is suitable for
testing) via curl or a browser with
http://solr_node:8983/solr/collection/update?commit=true

The reason we're focusing here is that:

Soft commits are disabled in your setup, this is the "-1" in autoSoftCommit.
Hard commits are not opening searchers, this is the autoCommit,
<openSearcher>false</openSearcher> section.

Are, you perhaps overriding the system variable
solr.autoSoftCommit.maxTime
when you start up Solr?

What about solr.autoCommit.maxTime (although this really shouldn't matter).

If you're not overriding the above, then no searchers should be being
opened at all after you start Solr, and only one should be opening
when you do start Solr. So you should not be getting the warning about
 Overlapping onDeckSearchers.

Forget the static warming queries, they are irrelevant until we
understand why you're getting any new searchers. For future reference,
these are the
newSearcher and firstSearcher events in solrconfig.xml. newSearcher is
fired every time one commits, firstSearcher when you start Solr.

The bottom line here is you need to find out why you're committing at
all, which opens a new searcher which, when that happens too often
generates the warning you're seeing.

Best,
Erick


On Wed, Aug 12, 2015 at 6:51 AM, Adrian Liew <adrian.liew@avanade.com> wrote:
> Hi Joel,
>
> I am fairly new to Solr (version which I am using v5.2.1) so I suppose what you may be
asking is referring to the autocommits section:
>
>
>     <!-- AutoCommit
>
>          Perform a hard commit automatically under certain conditions.
>          Instead of enabling autoCommit, consider using "commitWithin"
>          when adding documents.
>
>          http://wiki.apache.org/solr/UpdateXmlMessages
>
>          maxDocs - Maximum number of documents to add since the last
>                    commit before automatically triggering a new commit.
>
>          maxTime - Maximum amount of time in ms that is allowed to pass
>                    since a document was added before automatically
>                    triggering a new commit.
>          openSearcher - if false, the commit causes recent index changes
>            to be flushed to stable storage, but does not cause a new
>            searcher to be opened to make those changes visible.
>
>          If the updateLog is enabled, then it's highly recommended to
>          have some sort of hard autoCommit to limit the log size.
>       -->
>      <autoCommit>
>        <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>        <openSearcher>false</openSearcher>
>      </autoCommit>
>
>     <!-- softAutoCommit is like autoCommit except it causes a
>          'soft' commit which only ensures that changes are visible
>          but does not ensure that data is synced to disk.  This is
>          faster and more near-realtime friendly than a hard commit.
>       -->
>
>      <autoSoftCommit>
>        <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
>      </autoSoftCommit>
>
> Currently, this has been left with the defaults. Are you able to determine if I have
autocommits turned on or off?
>
> What do you mean by explicit commit? Does this mean a regular commit?
>
> Quick commits are fairly intermittent. Although, we will be running some automated tasks
to import some content which may directly cause frequent commits. This particular warning
does not happen often but does happen frequently within a one minute period. I predict with
automated import of content, this can trigger the warnings quite frequently. I have seen an
error reported back where the number of maxWarmingSearchers have been exceeded.
>
> In regards with checking to see if there are any static warm queries, how do you advise
which section to look at in the solrconfig.xml?
>
> Best regards,
>
> Adrian Liew
>
>
> -----Original Message-----
> From: Joel Bernstein [mailto:joelsolr@gmail.com]
> Sent: Wednesday, August 12, 2015 6:35 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Performance warning overlapping onDeckSearchers
>
> You'll want to check to see if there are any static warming queries being run as well.
>
> How often are you committing and opening a new searcher? Are you using autoCommits or
explicitly committing?
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Wed, Aug 12, 2015 at 3:57 AM, Adrian Liew <adrian.liew@avanade.com>
> wrote:
>
>> Additionally,
>>
>> I realized that my autowarmCount is set to zero for the following
>> Cache entries except perSegFilter:
>>
>>     <filterCache class="solr.FastLRUCache"
>>                  size="512"
>>                  initialSize="512"
>>                  autowarmCount="0"/>
>>
>>     <!-- Query Result Cache
>>
>>         Caches results of searches - ordered lists of document ids
>>         (DocList) based on a query, a sort, and the range of documents
>> requested.
>>         Additional supported parameter by LRUCache:
>>            maxRamMB - the maximum amount of RAM (in MB) that this
>> cache is allowed
>>                       to occupy
>>      -->
>>     <queryResultCache class="solr.LRUCache"
>>                      size="512"
>>                      initialSize="512"
>>                      autowarmCount="0"/>
>>
>>     <!-- Document Cache
>>
>>          Caches Lucene Document objects (the stored fields for each
>>          document).  Since Lucene internal document ids are transient,
>>          this cache will not be autowarmed.
>>       -->
>>     <documentCache class="solr.LRUCache"
>>                    size="512"
>>                    initialSize="512"
>>                    autowarmCount="0"/>
>>
>>     <!-- custom cache currently used by block join -->
>>     <cache name="perSegFilter"
>>       class="solr.search.LRUCache"
>>       size="10"
>>       initialSize="0"
>>       autowarmCount="10"
>>       regenerator="solr.NoOpRegenerator" />
>>
>>
>> The link,
>> https://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Ove
>> rlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F
>> did suggest to reduce the autowarmCount or reduce warm up cache
>> activity (which I am not sure where to begin doing this).
>>
>> I suspect autowarmCount is not very large as the above.
>>
>> Let me know what you think.
>>
>> Best regards,
>> Adrian Liew
>>
>>
>> -----Original Message-----
>> From: Adrian Liew [mailto:adrian.liew@avanade.com]
>> Sent: Wednesday, August 12, 2015 3:32 PM
>> To: solr-user@lucene.apache.org
>> Subject: RE: Performance warning overlapping onDeckSearchers
>>
>> Thanks Shawn. Having said that increasing maxWarmingSearchers is
>> usually wrong to solve this, are there any implications if we set
>> maxWarmingSearchers to zero to resolve this problem?
>>
>> Or do you think there are some other settings that are worthwhile
>> tuning to cater to the above?
>>
>> Best regards,
>> Adrian
>>
>> -----Original Message-----
>> From: Shawn Heisey [mailto:apache@elyograg.org]
>> Sent: Tuesday, August 11, 2015 11:02 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Performance warning overlapping onDeckSearchers
>>
>> On 8/11/2015 3:02 AM, Adrian Liew wrote:
>> > Has anyone come across this issue, [some_index] PERFORMANCE WARNING:
>> Overlapping onDeckSearchers=2?
>> >
>> > I am currently using Solr v5.2.1.
>> >
>> > What does this mean? Does this raise red flags?
>> >
>> > I am currently encountering an issue whereby my Sitecore system is
>> unable to update the index appropriately. I am not sure if this is
>> linked to the warnings above.
>>
>>
>> https://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Ove
>> rlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F
>>
>> What the wiki page doesn't explicitly state is that increasing
>> maxWarmingSearchers is usually the wrong way to solve this, because
>> that can actually make the problem *worse*.  It is implied by the
>> things the page DOES say, but it is not stated.
>>
>> Thanks,
>> Shawn
>>
>>

Mime
View raw message