lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Brennan <...@zimbra.com>
Subject Re: More IndexDeletionPolicy questions
Date Wed, 05 Mar 2008 22:31:28 GMT
Ha, you know it never occurred to me that the driver might do this for me...I'll test it out.
 Thanks,

--tim


----- "Michael McCandless" <lucene@mikemccandless.com> wrote:

> Oh then I don't think you need a custom deletion policy.
> 
> A single NFS client emulates delete-on-last-close semantics.  Ie, if 
> 
> the deletion of the file, and the held-open file handles, happen  
> through a single NFS client, then it emulates delete-on-last-close  
> semantics for you, by creating those .nfsXXX files on the server.
> 
> Test to be sure though!
> 
> Mike
> 
> Tim Brennan wrote:
> 
> > No, I have multiple readers in the same VM.  I track open readers  
> > within my VM and save those reader's commit points until the  
> > readers are gone.
> >
> > --tim
> >
> >
> >
> > ----- "Michael McCandless" <lucene@mikemccandless.com> wrote:
> >
> >> OK got it.  Yes, sharing an index over NFS requires your own
> >> DeletionPolicy.
> >>
> >> So presumably you have readers on different machines accessing the
> >> index over NFS, and then one machine that does the writing to that
> >> index?  How so you plan to get which commit point each of these
> >> readers is currently using back to the single writer?
> >>
> >> Mike
> >>
> >> Tim Brennan wrote:
> >>
> >>> The bigger picture here is NFS-safety.  When I run a search, I
> hand
> >>
> >>> off the search results to another thread so that they can be
> >>> processed as necessary -- in particular so that they can be
> JOINed
> >>
> >>> with a SQL DB -- but I don't want to completely lock the index
> from
> >>
> >>> writes while doing a bunch of SQL calls.  Using the commit point
> >>> tracking I can make sure my appropriate snapshot stays around
> until
> >>
> >>> I'm completely done with it, even if I'm using an NFS mounted
> >>> filesystem that doesn't have delete-last semantics.
> >>>
> >>> --tim
> >>>
> >>>
> >>>
> >>>
> >>>>> It seems like it should be pretty simple -- keep a list of open
> >>>>> IndexReaders, track what Segment files they're pointing to, and
> >> in
> >>>>
> >>>>> onCommit don't delete those segments.
> >>>>
> >>>> This implies you have multiple readers in a single JVM?  If so,
> >> you
> >>>> should not need to make a custom deletion policy to handle this
> >> case
> >>>>
> >>>> -- the OS should be properly protecting open files from deletion.
>  
> >>
> >>>> Can you  shed more light on the bigger picture here?
> >>>>
> >>>>> Unfortunately it ends up being very difficult to directly
> >> determine
> >>>>> what Segment an IndexReader is pointing to.  Is there some
> >>>>> straightforward way that I'm missing -- all I've managed to do
> so
> >>>>> far is to remember the most recent one from onCommit/onInit and
> >> use
> >>>>
> >>>>> that one....that works OK, but makes bootstrapping a pain if
> you
> >>>>> try to open a Reader before you've opened the writer once.
> >>>>>
> >>>>> Also, when I use IndexReader.reopen(), can I assume that the
> >> newly
> >>>>
> >>>>> returned reader is pointing at the "most recent" segment?  I
> >> think
> >>>>
> >>>>> so...
> >>>>
> >>>> Yes, except you have a synchronization challenge: if the writer
> is
> >> in
> >>>>
> >>>> the process of committing just as your reader opens you can't be
> >>>> certain whether the reader got the new commit or the previous
> one.
> >>>> If you have external synchronization to ensure reader only
> >> re-opens
> >>>> after writer has fully committed then this isn't an issue.
> >>>>
> >>>>>
> >>>>> Here's a sequence of steps that happen in my app:
> >>>>>
> >>>>> 0) open writer, onInit tells me that seg_1 is the most recent
> >>>> segment
> >>>>>
> >>>>> 1) Open reader, assume it is pointing to seg_1 from (0)
> >>>>>
> >>>>> 2) New write commits into seg_2, don't delete seg_1 b/c of (1)
> >>>>>
> >>>>> 3) Call reader.reopen() on the reader from (1)....new reader is
> >>>>> pointing to seg_2 now?
> >>>>>
> >>>>> 4) seg_1 stays around until the next time I open or commit a
> >>>>> writer, then it is removed.
> >>>>>
> >>>>>
> >>>>> Does that seem reasonable?
> >>>>>
> >>>>>
> >>>>> --tim
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: java-user-unsubscri
> >>>>
> >>>>
> >>>>
> >>
> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>> For additional commands, e-mail:
> java-user-help@lucene.apache.org
> >>>
> >>>
> >>
> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>
> >>
> >>
> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message