lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: default values for numRecordsToKeep and maxNumLogsToKeep
Date Thu, 20 Jul 2017 13:56:04 GMT
bq:  I am pretty sure that anytime a core starts for *any* reason, all
the transaction logs that are present will get replayed.

This isn't quite true. If Solr is shut down gracefully, or a hard
commit happened before shutdown (with no new docs added) then the tlog
will _not_ be replayed on startup. It's only when Solr is killed
without a commit and thus the tlog is not truncated (and segments not
closed) by hard commit that the tlog will be replayed on startup.
Which is why I strongly recommend that people stop Solr with the
script rather than "kill -9".

Best,
Erick

On Thu, Jul 20, 2017 at 5:39 AM, Shawn Heisey <apache@elyograg.org> wrote:
> On 7/18/2017 11:53 AM, suresh pendap wrote:
>> After looking at the source code I see that the default values for
>> numRecordsToKeep is 100 and maxNumLogsToKeep is 10.
>>
>> So it seems by default the replica can only have 1000 document updates lag
>> before the replica goes for a Full recovery from the leader.
>
> I don't think that's quite right.  In many situations, the number of
> documents in the transaction log will likely be less than 1000.
>
> Enough logs will be kept that *at least* 100 documents are there, if
> that can be accomplished with ten logfiles or less.  Based on a quick
> reading of the code, if the newest ten logs have less than 100
> documents, then there will be less than 100 docs available.  This would
> not end up being a problem for data integrity, because small infrequent
> updates would be the only way to end up with less than 100 docs, and in
> that situation, the small number of documents in the transaction log,
> when replayed at core startup, will be enough to ensure integrity.
>
> I think the reasons the default numbers are so small is an attempt to
> keep startup time low.  I am pretty sure that anytime a core starts for
> *any* reason, all the transaction logs that are present will get
> replayed.  I know for sure that this happens on Solr restart; I think it
> also happens on core reload.  By keeping the required minimum documents
> at a low value like 100, there's a better chance that the transaction
> logs will be small, and therefore core startup will be fast.
>
> On a system where there are no hard commits, all updates end up going
> into a single super-large transaction log.  This meets the default
> configuration numbers, because there are less than ten logs present, and
> what is present contains at least 100 documents.  Unfortunately, this
> means that when the core starts, it will replay that HUGE transaction
> log, a process that could take hours.  This situation is prevented by
> enabling autoCommit with a relatively short maxTime value.  Setting
> openSearcher to false in the autoCommit ensures that document visibility
> behavior is not altered by autoCommit.
>
> Thanks,
> Shawn
>

Mime
View raw message