lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <mmcc...@us.ibm.com>
Subject Re: using lucene Lock inter-jvm
Date Fri, 23 Jun 2006 13:25:21 GMT
OK, I did some digging on this ... using native locks has been
discussed before:

  * In this thread (2.5 yrs ago), to get locking across machines, a
    pluggable locking framework was created.  This looks very clean as
    it would allow, eg, "single JVM" instances of Lucene to use an
    in-process lock w/ FSDirectory instead of filesystem based locks.
    Locking is then independent of Directory implemenation in use:

 
http://www.gossamer-threads.com/lists/lucene/java-dev/24329?search_string=pluggable%20lock;#24329

    This was then opened as an issue with nice patch:

      http://issues.apache.org/jira/browse/LUCENE-305

  * Also 2.5 yrs ago, this bug (separately) was opened, which looks
    like a starvation issue (because there's no "scheduling" of lock
    acquisitions), with a patch was that uses native locks:

      http://issues.apache.org/jira/browse/LUCENE-307

  * Then in this thread (2 yrs ago), native locks were explored as a
    way to make locking over NFS work, but apparently (??) didn't
    reach closure because the approach seemed to hang over NFS:

 
http://www.gossamer-threads.com/lists/lucene/java-dev/22464?search_string=FileLock;#22464

  * There in this thread (1.5 yrs ago) it seems this was the original
    driver for switching from 1.3 to 1.4 (but then didn't happen):

 
http://www.gossamer-threads.com/lists/lucene/java-dev/25034?search_string=FileLock;#25034

It seems like:

  * Locking is a sore point now.  Just googling on the "lock obtain
    timed out" produces quite a few results.

  * Most of these are probably from hard shutdowns (jvm crashed or was
    killed, power lost to machine, etc) leaving lock files in the
    filesystem, which native locking would automatically prevent.

  * Some of these cases are probably the thread starvation case
    (LUCENE-307) maybe (speculating...) from frequent interleaving of
    deleting docs via IndexReader & adding docs via IndexWriter.  With
    the right Lock class we can resolve this.

  * Remote locking is a relatively common need but not supported now
    (and native locking and/or pluggable locking w/ a DB or other
    implementation, should be able to resolve).

I think the best course of action here is:

  * Let's first try to get the patches in LUCENE-305 updated to
    current HEAD sources and then committed?  This can be done
    independent of native locks, by having the current locking
    implementation be the default Lock class for starters.

  * Next, create a native locking class that extends Lock.  Borrow /
    copy / be inspired from the examples above (and also from Hadoop /
    Jackrabbit and others).  Try to resolve other issues (the
    starvation issue above; using wait/notify instead of sleep /
    poll).  Add this to core or contrib, and at some point make it the
    default Lock implementation for FSDirectory.

Does that sound right?  If so, how to nudge this forward?  There was
thread recently on voting for bugs to fix.  I vote +1 for LUCENE-305!

I'll volunteer to update LUCENE-305 to current HEAD unless original
author (or someone else) wants to?

Mike




"Yonik Seeley" <yseeley@gmail.com> 
06/21/2006 11:53 AM
Please respond to
java-dev@lucene.apache.org


To
java-dev@lucene.apache.org
cc
java-user@lucene.apache.org
Subject
Re: using lucene Lock inter-jvm






On 6/21/06, Michael McCandless <mmccand@us.ibm.com> wrote:
> Does anyone know of any reasons not to switch Lucene's FSDirectory 
locking
> to the java.nio.channels.FileLock?  EG, are there any performance issues
> that people are aware of?  It's available since Java 1.4.

Good question Michael, no reason that I know of... I think its
probably just that no one has revisited the issue since Lucene moved
to 1.4

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search 
server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message