lucene-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ilan Ginzburg (Jira)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-13932) Review directory locking for Blob interactions
Date Sun, 24 Nov 2019 20:30:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-13932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16981175#comment-16981175
] 

Ilan Ginzburg commented on SOLR-13932:
--------------------------------------

As a follow up to [the first comment on this Jira|https://issues.apache.org/jira/browse/SOLR-13932?focusedCommentId=16974191&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16974191],
Blob metadata does not need to track the index generation and local index generation does
not have to be compared to the Blob generation. Decision to *pull* from Blob should only be
based on differences between local commit point and Blob set of files (if the two are different,
Blob is the source of truth). Decision to *push* to Blob should only be made when local index
was up to date with Blob, an update occurred locally and we verify that no other replica (on
this or on another node) updated the Blob files in the meantime.

Therefore, as part of this Jira I'm going to remove the generation member variable in {{BlobCoreMetadata}},
{{BlobCoreMetadataBuilder}} and the various places where it is then used. It is used for resolution
in {{SharedStoreResolutionUtil}} and elsewhere for logging.

I believe the checks described in [my previous post above|https://issues.apache.org/jira/browse/SOLR-13932?focusedCommentId=16977555&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16977555]
are sufficient to make sure we avoid conflicts between local and pulled files regardless of
which generation each index is at.


> Review directory locking for Blob interactions
> ----------------------------------------------
>
>                 Key: SOLR-13932
>                 URL: https://issues.apache.org/jira/browse/SOLR-13932
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Ilan Ginzburg
>            Priority: Major
>
> Review resolution of local index directory content vs Blob copy.
> There has been wrong understanding of following line acquiring a lock on index directory.
>  {{solrCore.getDirectoryFactory().get(indexDirPath, DirectoryFactory.DirContext.DEFAULT,
solrCore.getSolrConfig().indexConfig.lockType);}}
> From Yonik:
> _A couple things about Directory locking.... the locks were only ever to prevent more
than one IndexWriter from trying to modify the same index. The IndexWriter grabs a write lock
once when it is created and does not release it until it is closed._ 
> _Directories are not locked on acquisition of the Directory from the DirectoryFactory.
See the IndexWriter constructor, where the lock is explicitly grabbed._
> Review CorePushPull#pullUpdateFromBlob, ServerSideMetadata and other classes as relevant.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Mime
View raw message