lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <apa...@elyograg.org>
Subject Re: solrcloud Auto-commit doesn't seem reliable
Date Fri, 09 Feb 2018 21:49:00 GMT
On 2/9/2018 9:29 AM, Webster Homer wrote:
> A little more background. Our production Solrclouds are populated via CDCR,
> CDCR does not replicate commits, Commits to the target clouds happen via
> autoCommit settings
> 
> We see relvancy scores get inconsistent when there are too many deletes
> which seems to happen when hard commits don't happen.
> 
> On Fri, Feb 9, 2018 at 10:25 AM, Webster Homer <webster.homer@sial.com>
> wrote:
> 
>> I we do have autoSoftcommit set to 3 seconds. It is NOT the visibility of
>> the records that is my primary concern. I am concerned about is the
>> accumulation of uncommitted tlog files and the larger number of deleted
>> documents.

For the deleted documents:  Have you ever done an optimize on the 
collection?  If so, you're going to need to re-do the optimize regularly 
to keep deleted documents from growing out of control.  See this issue 
for a very technical discussion about it:

https://issues.apache.org/jira/browse/LUCENE-7976

Deleted documents probably aren't really related to what we've been 
discussing.  That shouldn't really be strongly affected by commit settings.

-----

A 3 second autoSoftCommit is VERY aggressive.   If your soft commits are 
taking longer than 3 seconds to complete, which is often what happens, 
then that will lead to problems.  I wouldn't expect it to cause the 
kinds of problems you describe, though.  It would manifest as Solr 
working too hard, logging warnings or errors, and changes taking too 
long to show up.

Assuming that the config for autoSoftCommit doesn't have the typo that 
Erick mentioned.

----

I have never used CDCR, so I know very little about it.  But I have seen 
reports on this mailing list saying that transaction logs never get 
deleted when CDCR is configured.

Below is a link to a mailing list discussion related to CDCR not 
deleting transaction logs.  Looks like for it to work right a buffer 
needs to be disabled, and there may also be problems caused by not 
having a complete zkHost string in the CDCR config:

http://lucene.472066.n3.nabble.com/CDCR-how-to-deal-with-the-transaction-log-files-td4345062.html

Erick also mentioned this.

Thanks,
Shawn

Mime
View raw message