lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: lucene on amazon s3
Date Wed, 02 Sep 2009 10:57:59 GMT
So long as you can ensure, external to Lucene, that only one
IndexWriter is open at once on the index, you can disable all of
Lucene's normal locking.  But you must be certain: if you accidentally
allow two IndexWriter's to be open at once, it will quickly corrupt
the index.

Beyond locking, Lucene also generally requires client side cache
coherence of the filesystem.  Notably, NFS and the 2.0 version of
Microsoft's SMB/CIFS (the default in Windows Server 7 & Vista), have
cache coherence problems (SMB/CIFS 1.0 works fine).  EG, if a writer
somewhere updates the index, when a reader somewhere else lists the
directory, is it guaranteed that the directory listing is always

If there are cache coherence problems, it usually manifests as a
"false" FileNotFoundException... the good news is, you can catch this,
wait a bit, and retry, as a workaround.

If you get any data on S3's cache coherence, please post back!


On Wed, Sep 2, 2009 at 5:55 AM, prasenjit<> wrote:
> I am exploring the possibility of creating large lucene indices via ec2/s3.
> Till now I have got only teh following url :
> But still dont know whether the lucene locking problem ( on a distributed FS
> like S3/DFS ) is fixed or not. Any information is greatly apreciated.
> --
> View this message in context:
> Sent from the Lucene - Java Users mailing list archive at
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message