lucene-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ilan Ginzburg (Jira)" <>
Subject [jira] [Commented] (SOLR-13932) Review directory locking for Blob interactions
Date Thu, 14 Nov 2019 12:26:00 GMT


Ilan Ginzburg commented on SOLR-13932:

Commented code in {{ServerSideMetadata}} constructor regarding {{DirectoryReader.listCommits()}}
mentions "But in blob context we serialize commits and pulls by proper locking therefore we
should be good here". I believe this is no longer true (a previous Salesforce version of that
code used to serialize updates). We do not serialize commits but let Lucene do it the way
it wants (we serialize pushes to Blob).

OTOH, I don't see why we try to detect inconsistency using past files that are no longer part
of the commit point. If we rely on such detections, we're at risk because files of past commit
points can be deleted at any time. Our strategy must make sure (I believe it does) that local
is always consistent, or local should be deleted and the Blob version adopted instead. The
only time where local is "a source of truth" over blob is after indexing. Consistency in such
a case is guaranteed by verifying that local was consistent with Blob before indexing started
(blob being source of truth), and that Blob hasn't changed by the time indexing finished and
got pushed back to Blob (allowing parallel indexing from given node/replica, this is done
by the conditional update into ZK).

> Review directory locking for Blob interactions
> ----------------------------------------------
>                 Key: SOLR-13932
>                 URL:
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Ilan Ginzburg
>            Priority: Major
> Review resolution of local index directory content vs Blob copy.
> There has been wrong understanding of following line acquiring a lock on index directory.
>  {{solrCore.getDirectoryFactory().get(indexDirPath, DirectoryFactory.DirContext.DEFAULT,
> From Yonik:
> _A couple things about Directory locking.... the locks were only ever to prevent more
than one IndexWriter from trying to modify the same index. The IndexWriter grabs a write lock
once when it is created and does not release it until it is closed._ 
> _Directories are not locked on acquisition of the Directory from the DirectoryFactory.
See the IndexWriter constructor, where the lock is explicitly grabbed._
> Review CorePushPull#pullUpdateFromBlob, ServerSideMetadata and other classes as relevant.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message