lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Adding DocExpirationUpdateProcessorFactory causes "Overlapping onDeckSearchers" warnings
Date Fri, 09 Dec 2016 23:39:13 GMT
bq: ...is it triggering a commit every 30 seconds, because that's what
I have the autoDeletePeriodSeconds set to


Yep. There's this line from Chris' writeup:
****
After the deleteByQuery has been executed, a soft commit is also
executed using openSearcher=true so that search results will no longer
see the expired documents.
****

Assuming your index is changing, you'll indeed open one searcher as a
result of your autocommit and a second as a result of TTL processing.
And they'll overlap sometimes.

There's a note in the code about making the commits optional, it seems
fair to raise a JIRA about implementing this. Patches even more
welcome ;).

Meanwhile, this performance _warning_ is benign. That is, the new
searcher is indeed opened. If you see something like "Error opening
new searcher. exceeded limit of maxWarmingSearchers...." then the
newest searcher will not be opened (although the next searcher opened
will pick up any changes).

Note that there's no particular point in bumping the max warming
searchers in solrconfig.xml since the warning message is dumped
whenever there are > 1 warming searchers. If you get the "error
opening...." message it's a more open question though.

So your choices are:
1> just ignore it
2> contribute a patch
3> increase the interval. That'll reduce the number of times you see
this warning, but won't eliminate them all.

Best,
Erick


On Fri, Dec 9, 2016 at 3:15 PM, Brent <brent.pearson@gmail.com> wrote:
> I'm using Solr Cloud 6.1.0, and my client application is using SolrJ 6.1.0.
>
> Using this Solr config, I get none of the dreaded "PERFORMANCE WARNING:
> Overlapping onDeckSearchers=2" log messages:
> https://dl.dropboxusercontent.com/u/49733981/solrconfig-no_warnings.xml
>
> However, I start getting them frequently after I add an expiration update
> processor to the update request processor chain, as seen in this config (at
> the bottom):
> https://dl.dropboxusercontent.com/u/49733981/solrconfig-warnings.xml
>
> Do I have something configured wrong in the way I've tried to add the
> function of expiring documents? My client application sets the "expire_at"
> field with the date to remove the document being added, so I don't need
> anything on the Solr Cloud side to calculate the expiration date using a
> TTL. I've confirmed that the documents are getting removed as expected after
> the TTL duration.
>
> Is it possible that the expiration processor is triggering additional
> commits? Seems like the warning is usually the result of commits happening
> too frequently. If the commit spacing is fine without the expiration
> processor, but not okay when I add it, it seems like maybe each update is
> now triggering a (soft?) commit. Although, that'd actually be crazy and I'm
> sure I'd see a lot more errors if that were the case... is it triggering a
> commit every 30 seconds, because that's what I have the
> autoDeletePeriodSeconds set to? Maybe if I try to offset that a bit from the
> 10 second auto soft commit I'm using? Seems like it'd be better (if that is
> the case) if the processor simple didn't have to do a commit when it expires
> documents, and instead let the auto commit settings handle that.
>
> Do I still need the line:
> <requestHandler class="solr.UpdateRequestHandler"
> name="/update"></requestHandler>
> when I have the
> <updateRequestProcessorChain name="doc-expiration-processor-chain"
> default="true">
> element?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Adding-DocExpirationUpdateProcessorFactory-causes-Overlapping-onDeckSearchers-warnings-tp4309155.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message