lucene-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ilan Ginzburg (Jira)" <>
Subject [jira] [Commented] (SOLR-13932) Review directory locking for Blob interactions
Date Sun, 24 Nov 2019 21:29:00 GMT


Ilan Ginzburg commented on SOLR-13932:

As part of Blob being "source of truth", code in {{CorePusher.pushCoreToBlob()}} that compares
the generation of local commit point to the Blob generation should go away. As stated previously,
the decision to push should not be based on generation.
As a matter of fact, we can ignore the notion of generation altogether in the Blob push/pull
code. All that really matters is do we have the latest copy from Blob, and once we do and
we update it locally, while pushing we should make sure no other conflicting update happened
to Blob, to make sure updates can't overwrite each other.

Also, the push strategy currently takes a snapshot i.e. copies all the files in the current
commit point into a new directory (parameter {{takeSnapshot}} in the {{ServerSideMetadata}}
constructor) to push them from there. This is used by {{CorePusher.pushCoreToBlob()}} and
could lead to inefficient push operations. Pushes should instead be done from the existing
index directory without making copies.
{{ReplicationHandler}} does not do a copy and does its best to reserver the commit point while
the client is replicating files. The {{IndexFetcher}} has to deal with Exceptions when getting
the file stream (see for example {{IndexFetcher.downloadIndexFiles()}}).

> Review directory locking for Blob interactions
> ----------------------------------------------
>                 Key: SOLR-13932
>                 URL:
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Ilan Ginzburg
>            Priority: Major
> Review resolution of local index directory content vs Blob copy.
> There has been wrong understanding of following line acquiring a lock on index directory.
>  {{solrCore.getDirectoryFactory().get(indexDirPath, DirectoryFactory.DirContext.DEFAULT,
> From Yonik:
> _A couple things about Directory locking.... the locks were only ever to prevent more
than one IndexWriter from trying to modify the same index. The IndexWriter grabs a write lock
once when it is created and does not release it until it is closed._ 
> _Directories are not locked on acquisition of the Directory from the DirectoryFactory.
See the IndexWriter constructor, where the lock is explicitly grabbed._
> Review CorePushPull#pullUpdateFromBlob, ServerSideMetadata and other classes as relevant.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message