lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Preventing multiple on-deck searchers without causing failed commits
Date Wed, 19 Feb 2014 03:05:34 GMT
Colin:

Stop. Back up. The automatic soft commits will make updates available to
your users every second. Those documents _include_ anything from your "hard
commit" jobs. What could be faster? Parenthetically I'll add that 1 second
soft commits are rarely an actual requirement, but that's your decision.

For the hard commits. Fine. Do them if you insist. Just set
openSearcher=false. The documents will be searchable the next time the soft
commit happens, within one second. The key is openSearcher=false. That
prevents starting a brand new searcher.

BTW, your commits are not failing. It's just that _after_ the commit
happens, the warming searcher limit is exceeded.

You can even wait until the segments are flushed to disk. All without
opening a searcher.

Shawn is spot on in his recommendations to not fixate on the commits. Solr
handles that. Here's a long blog about all the details of durability .vs.
visibility.
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

You're over-thinking the problem here, trying to control commits with a
sledgehammer when you don't need to, just use the built-in capabilities.

Best,
Erick



On Tue, Feb 18, 2014 at 10:33 AM, Colin Bartolome <colin@e-e.com> wrote:

> On 02/18/2014 10:15 AM, Shawn Heisey wrote:
>
>> If you want to be completely in control like that, get rid of the
>> automatic soft commits and just do the hard commits.
>>
>> I would personally choose another option for your setup -- get rid of
>> *all* explicit commits entirely, and just configure autoCommit and
>> autoSoftCommit in the server config.  Since you're running 4.x, you really
>> should have the transaction log (updateLog in the config) enabled.  You
>> can rely on the transaction log to replay updates since the last hard
>> commit if there's ever a crash.
>>
>> I would also recommend upgrading to 4.6.1, but that's a completely
>> separate item.
>>
>> Thanks,
>> Shawn
>>
>>
> We use the automatic soft commits to get search index updates to our users
> faster, via Near Realtime Searching. We have the updateLog enabled. I'm not
> worried that the Solr side of the equation will lose data; I'm worried that
> the communication from our web servers and scheduled jobs to the Solr
> servers will break down and nothing will come along to make sure everything
> is up to date. It sounds like what we're picturing is not currently
> supported, so I'll file the RFE.
>
> Will upgrading to 4.6.1 help at all with this issue?
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message