lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: SMB2 cache
Date Thu, 13 Aug 2009 21:50:42 GMT
On Thu, Aug 13, 2009 at 5:33 PM, Shai Erera<> wrote:

> So if afterwards we read until segment_17 and exhaust read-ahead, and we
> determine that there's a problem - we throw the exception. If instead we'll
> try to read backwards, I'm sure one of the segments will be read
> successfully, because that reader must already see any segment, right?

I don't think you're guaranteed to read successfully, on reading backwards.

Ie, say writer has committed segments_8, and therefore just removed segments_7.

When the reader (on a different machine, w/ stale cache) tries to
open, it's cache claims segments_7 still exists, so we try to open
that but fail.  We advance to segments_8 and try to open that, but
fail (presumably because local SMB2 cache doesn't consult the server,
unlike many NFS clients, I think).  We then try up through segments_17
and nothing works.  But going backwards can't work either because
those segments files have all been deleted.  (Assuming
KeepOnlyLastCommitDeletionPolicy... things do get more interesting if
you're using a different deletion policy...).

Sadly, the most common approach to refreshing readers, eg checking
every N seconds if it's time to reopen, leads directly to this "cache
is holding onto stale data".  My guess is if an app only attempted to
reopen the reader after the writer on another machine had committed,
then this exception wouldn't happen.  But that'd require some
notification mechanism outside of Lucene.


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message