lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shai Erera <ser...@gmail.com>
Subject Re: SMB2 cache
Date Thu, 13 Aug 2009 21:02:12 GMT
What about doing a read-backwards as well? I.e., if read ahead fails, try to
read backwards --> we must be able to read any segment, no?

Shai

On Thu, Aug 13, 2009 at 9:29 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> This is spooky -- it looks like SMB2 (which was introduced with Windows
> Vista & Windows Server 2008) now does "aggressive" client-side
> caching, such that the cache can be wrong about the current state of
> the directory.
>
> At least it sort of sounds like Microsoft considers it a real issue:
>
> > Yes, this is a known product issue of SMB2.
> >
> > SMB2 does implicit attribute and directory metadata caching at all
> > times, whereas SMB1 was much stricter about when it would do so. The
> > caches are consistent when changes are made by the client, but if
> > changes are made from another client they may not be reflected until
> > the cache times out.
>
> This will definitely cause problems (like the exception you're
> hitting) for Lucene.  It's exactly the same problems we had with NFS,
> but the readahead in SegmentInfos.FindSegmentsFile worked around that.
> It sounds like for SMB2 that readahead is not working, presumably
> because (unlike NFS) the client does not check back w/ the server if
> it believes (based on its stale cache) that the file does not exist.  Sigh.
>
> SMB1 did not have this problem, in my experience.
>
> I wonder if, from javaland, we have some way to force the cache to
> become coherent.
>
> One simple workaround at the app level is to simply retry on hitting
> an errant "segments_N file not found" exception.
>
> Mike
>
> On Thu, Aug 13, 2009 at 8:50 AM, Shai Erera<serera@gmail.com> wrote:
> > Hi
> >
> > Has anyone experienced any problems w/ Lucene indexes on a shared SMB2
> > network drive?
> >
> > We've hit a scenario where it seems the FS cache refuses to check for
> > existence of files on the shared network drive. Specifically, we hit the
> > following exception:
> >
> > java.io.FileNotFoundException: Z:\index\segments_p8 (The system cannot
> find
> > the file specified.)
> > at java.io.RandomAccessFile.open(Native Method)
> > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
> > at
> >
> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
> > at
> >
> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
> > at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
> > at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:482)
> > at org.apache.lucene.index.SegmentInfos$2.doBody(SegmentInfos.java:369)
> > at
> >
> org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
> > at
> >
> org.apache.lucene.index.SegmentInfos.readCurrentVersion(SegmentInfos.java:366)
> > at
> >
> org.apache.lucene.index.DirectoryIndexReader.isCurrent(DirectoryIndexReader.java:188)
> > at org.apache.lucene.index.MultiReader.isCurrent(MultiReader.java:352)
> >
> > The environment:
> > * 3 Windows Server 2008 machines
> > ** Machine A - hosts the index
> > ** Machine B - indexes and search
> > ** Machine C - just search
> > * Machine A and B map Machine C on drive Z.
> > * The exception happens on Machine C only, i.e. on the machine that does
> > just 'search'.
> >
> > According to my understanding, FindSegmentFile attempts to read the
> latest
> > segment from segments.gen and directory listing and if there is a
> problem,
> > it will do a gen-readahead until success or defaultGenLookaheadCount is
> > exhausted.
> >
> > So by hitting this exception we thought of the following explanation: the
> FS
> > cache 'decides' the file does not exist, due to a stale directory cache,
> and
> > refuses to check whether the file actually exists on the remote machine.
> >
> > Does that sound reasonable?
> >
> > Some more information:
> > * We use Lucene 2.4.0
> > * Other runs are executed on those machines currently, and so it will
> take
> > about a week until we can run the same scenario again. I thought that
> > perhaps we can discuss this until then.
> > * Unfortunately we weren't able to get an infoStream output before the
> > machines started another run, so we hope to get it next time. Anyway,
> it's
> > not easily reproduced.
> > * There isn't any other process which touches this directory, such that
> it
> > may remove index files.
> >
> > We know the same code runs well on NFS (4). We haven't checked yet if SMB
> > 1.0 works ok. Some pointers we've found:
> >
> > A known issue on MS, w/ some C++ fixes:
> >
> http://www.microsoft.com/communities/newsgroups/en-us/default.aspx?dg=microsoft.public.win32.programmer.networks&tid=69e63e38-7d91-4306-ab6e-a615e1c6afaa&cat=en_US_bc89adf4-f184-4d3d-aaee-122567385744&lang=en&cr=US&sloc=&p=1
> >
> > Info on how to disable SMB 2.0 on Windows:
> >
> http://www.petri.co.il/how-to-disable-smb-2-on-windows-vista-or-server-2008.htm
> >
> > Currently, we think to bypass the problem by wrapping calls to isCurrent
> and
> > reopen w/ a try-catch FileNotFoundException and use the reader we have at
> > hand. Later, we will attempt the isCurrent again. Since SMB caching seems
> to
> > be time-controlled, we expect the cache to be refreshed after several
> > seconds, and those calls will succeed.
> > I wonder though if this can't get us into hitting the exception
> 'forever'.
> > E.g., imagine a system which indexes at very high rates. Isn't it
> possible
> > that we'll hit this exception every time we call isCurrent?
> >
> > I'm not sure if there is anything we can do in Lucene, besides sleeping
> in
> > FindSegmentsFile for several seconds which is not reasonable.
> > Maybe a way out would be, I think, having FindSegmentsFile try to read
> ahead
> > and then backwards. At some point, we ought to find a segment that's
> > readable, even if an old one, no?
> >
> > Any help will be appreciated.
> >
> > Shai
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

Mime
View raw message