lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pete Lewis" <>
Subject Re: Lock handling and Lucene 1.9 / 2.0
Date Mon, 13 Sep 2004 22:07:21 GMT
Hi Christoph

The directory caching is applied *across* class instances (the directory
is instanced once) - this cache exists singularily and is updated if the
FSDirectory is called against a different index.

Multiple indexes will *always* cause directory caching upon calls to
FSDirectory - our searches are made sequentially against all libraries
(or a selection of libraries) and this sequential call to FSDirectory
causes the cache to be updated - its very, very rare that the cache will
remain the same between two calls to get FSDirectories. This caching
*is* synchronized using the commit.lock (see the code) and two processes
(independent JVM's) will attain two different caches (completely
separate) *but* are tied together by the commit lock. This is what
causes the spin.

> FSDirectory.getDirectory has nothing to do with a commit.lock!

Err, wrong. The directory.makeLock(IndexWriter.COMMIT_LOCK_NAME) call
from within the routine ties the commit.lock to the
FSDirectory by synchronising the code around a *static* instance of the
directory object (see the code!!).

Pete Lewis

----- Original Message ----- 
From: "Christoph Goller" <>
To: "Lucene Developers List" <>
Sent: Monday, September 13, 2004 11:34 AM
Subject: Re: Lock handling and Lucene 1.9 / 2.0

> Pete Lewis wrote:
> > Hi Christoph
> > Long answer - theres a heap of horrible, horrible code in the
FSDirectory that tries to be clever and I think its not quite working
> >
> > Two types of lock - write.lock and commit.lock. The write.lock is used
exclusively for synchronising the indexing of documents and has *no* impact
on searching whatsoever.
> >
> > Commit.lock is another little story. Commit.lock is used for two
things - stopping indexing processes from overwriting segments that another
one is currently using, and stopping IndexReaders from overwriting each
other when they delete entries (dcon't even start asking my why a bloody
IndexReader can delete documents).
> Commit.lock is used to synchronize comittment of changes to an index
> with the process of opening an IndexReader. These changes my come from
> an IndexWriter or an IndexReader. There are good reasons for having the
> delete functionality in IndexReader (see developer mailing list around
> July 16). Write.lock is used to gurantee that there always is only one
> writer.
> >
> > *However*, theres another naughty little usage that isn't listed in any
of the documentation, and here it is....
> >
> > Doug Cutting wrote FSDirectory in such a way that it caches a directory.
Hence, if FSDirectory is called more than once with the same directory, the
FSDirectory class uses a static Hashtable to return the current values.
However, if FSDirectory is called with a *different* directory, it engages a
commit.lock while it updates the values. It *also* makes that Hashtable
> FSDirectory.getDirectory has nothing to do with a commit.lock!
> Lucene currently uses 2 locking mechanisms, the interprocess
> mechanism with the commit.lock file and an intraprocess mechanism
> based on synchronization on directory instances. The 2nd mechanism
> needs unique directory instances and this is achieved by caching
> directory instances in FSDirectory.
> >
> > Creating an IndexSearcher creates (within itself) an IndexReader to read
the index. The first thing the IndexReader does is grab an FSDirectory for
the index directory - if you are using LUCENE with a single index, theres is
never a problem - it is read once, then cached.
> >
> > Our search process works by searching across all the libraries selected
sequentially, building a results list and then culling the results it
doesn't need. To search it loops through each library and creates an
IndexSearcher to get at the data.
> >
> > Starting to see the issue yet? Because each library is in a different
directory, the internal call to the IndexReader which then gets an
FSDirectory causes the FSDirectory to update its singular cache. Which
forces a commit.lock to appear.
> >
> > Doug Cuttings little bit of 'neat' code for caching singularily the data
within an FSDirectory is causing us headaches immense. The code is horrible:
> >
> > /** Returns an IndexReader reading the index in the given Directory. */
> >   public static IndexReader open(final Directory directory) throws
> >     synchronized (directory) {     // in- & inter-process sync
> >       return (IndexReader)new Lock.With(
> >           directory.makeLock(IndexWriter.COMMIT_LOCK_NAME),
> >           IndexWriter.COMMIT_LOCK_TIMEOUT) {
> >           public Object doBody() throws IOException {
> >             SegmentInfos infos = new SegmentInfos();
> >   ;
> >             if (infos.size() == 1) {    // index is optimized
> >               return new SegmentReader(infos,, true);
> >             } else {
> >                 SegmentReader[] readers = new
> >                 for (int i = 0; i < infos.size(); i++)
> >                   readers[i] = new SegmentReader(infos,,
> >                 return new SegmentsReader(infos, directory, readers);
> >             }
> >           }
> >         }.run();
> >     }
> >   }
> >
> > Where directory is passed in from the constructor to IndexReader thus:
> >
> >   return open( FSDirectory.getDirectory( path, false ) );
> All threads that open an IndexReader and that don't get a directory
> directly have to compete for FSDirectory.getDirectory synchronization
> independent of the index you are trying to open. So you are right. This
> is a bottleneck.
> After that, threads opening an IndexReader only compete with each other
> if they try to read the same index. This is handled by the two above
> mentioned locking mechanisms.
> Here are two ideas that could help:
> The bottleneck only occurs if you always start a new process for every
> doesn't it? If you make a second search within the same process,
> the directory instances will already be cached and the bottleneck won't be
> problem? Furthermore, you do not have to always open new searchers for
> search. Can't you use your Searcher instances for multiple searches.
> A question for Lucene 1.9/2.0 is, whether we really need intraprocess and
> interprocess synchonization. Maybe these two mechanisms exist for purely
> historical reasons and the interprocess mechanism alone would be enough?
> Christoph
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message