Mailing-List: contact lucene-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
Received-SPF: pass (hermes.apache.org: local policy)
Message-ID: <20040915000454.61956.qmail@web12702.mail.yahoo.com>
Date: Tue, 14 Sep 2004 17:04:54 -0700 (PDT)
From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
Subject: Re: Lock handling and Lucene 1.9 / 2.0
To: Lucene Developers List <lucene-dev@jakarta.apache.org>
In-Reply-To: <114101c49a93$1d987fe0$34c2e550@joseph>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Hello Pete,

--- Pete Lewis <pete@uptima.co.uk> wrote:

> Hi Otis
> 
> The only way to continually use the same IndexReader would be to use
> a
> stateful session bean (frowned upon by J2EE Container writers), and
> then
> we'd see the same problem cross-machine - remember that the indexes
> that we
> use are stored once, not replicated across the four servers.
> 
> We'll come up with a fix ourselves.

Excellent.  Please contribute it back, if you can.

> I thought that it might be a good candidate for Lucene 2 as the
> FSDirectory code is horrible and non-J2EE compliant.

Patches are welcome!  As you saw on the Wiki, we (well, the German part
of the development team, really) are making bigger changes than usual,
including a deprecation of some frequently used pieces of the API. 
Including a code improvement for version 1.9/2.0 shouldn't be a problem
then.

> Is there any place for making suggestions for future development
> tasks?
> I've seen the Lucene 2 whiteboard and wondered how the items on the
> list
> were derived.

Discussion on lucene-dev => agreement => conclusion => Wiki.  I think
you need an account for Wiki now (pain, but thank spammers for that).

Otis

> Pete Lewis
> 
> 
> ----- Original Message ----- 
> From: "Otis Gospodnetic" <otis_gospodnetic@yahoo.com>
> To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
> Sent: Tuesday, September 14, 2004 12:11 PM
> Subject: Re: Lock handling and Lucene 1.9 / 2.0
> 
> 
> > It sounds like this is the source of your problem.  Reuse those
> > IndexReaders or IndexSearchers and you'll avoid the waits you are
> > talking about, as Christoph already pointed out.  How you implement
> the
> > logic for caching IndexReaders/Searchers is up to you.
> >
> > Otis
> > P.S.
> > It sounds like you took the time to figure out how some of the
> Lucene
> > internals work.  The best way to persuade -dev list subscribers
> that
> > doing something one way is better than doing it the current way is
> by
> > providing a clean diff against CVS HEAD, attached to an entry in
> > Bugzilla.  There are now several active Lucene developers and
> > contributors who will look at your suggestion and apply the patch,
> if
> > it improves the code.
> >
> >
> >
> > --- Pete Lewis <pete@uptima.co.uk> wrote:
> >
> > > Hi Christoph
> > >
> > > We are in a cluster running under Bea Weblogic.
> > >
> > > We have a static API to the search component from the portlets.
> > >
> > > We therefore need to open an indexreader per request.
> > >
> > > Cheers
> > > Pete
> > >
> > > ----- Original Message ----- 
> > > From: "Christoph Goller" <goller@detego-software.de>
> > > To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
> > > Sent: Tuesday, September 14, 2004 10:55 AM
> > > Subject: Re: Lock handling and Lucene 1.9 / 2.0
> > >
> > >
> > > > Pete Lewis wrote:
> > > > > Hi Christoph
> > > > >
> > > > > If we stand back a second and ask why we have commit locks
> when
> > > searching?
> > > > >
> > > > > The answer comes down to handling the FSDirectory - where the
> > > methods
> > > used
> > > > > are not j2ee compliant.
> > > > >
> > > > > We could open another can of worms and say why does the
> > > indexreader
> > > delete -
> > > > > but I won't go into that argument again here.....
> > > > >
> > > > > The bottom line is that we need the ability to search without
> > > waiting on
> > > a
> > > > > commit lock.
> > > >
> > > > You have this ability already !!!!!!!!!!!!!!!!!!!!!!!!!
> > > >
> > > > The FSDirectory is where the problems lie.  We could hack the
> > > > > code to make it work for our particular application; however
> what
> > > I've
> > > been
> > > > > trying to get across is the need to have a method that will
> give
> > > users
> > > the
> > > > > capability to just search (not delete) without waiting upon
> the
> > > commit
> > > lock,
> > > > > that will be j2ee compliant, and that will be appropriate
> > > clustered
> > > > > implementations - and that this should be a candidate for
> Lucene
> > > 1.9 /
> > > 2.0.
> > > > >
> > > > > You say that it shouldn't take long to wait.  A 1 sec spin
> lock
> > > per
> > > index
> > > > > per process is an eternity when trying to scale for
> potentially
> > > thousands of
> > > > > users.
> > > >
> > > > I have to admit that I am not an expert in j2ee compliancy. But
> I
> > > would
> > > like
> > > > to learn about it. If a database (I consder Lucene as a
> database)
> > > really
> > > has
> > > > to be initialized for every read-access, than there is a
> problem
> > > with j2ee
> > > > compliancy. I cannot believe that this is really true.
> > > >
> > > > LET ME STATE AGAIN: You should not open a new IndexReader for
> every
> > > > search/query. If you do so you definitely have a performance
> > > problem
> > > > independently from synchronization!!!!!!!!! Opening an
> IndexReader
> > > is
> > > > much more expensive than any individual query/search.
> > > >
> > > > You need a commit.lock for opening an IndexReader not because
> > > IndexReader
> > > > has delete functionality. You need it because if there is some
> > > process
> > > > writing to your index, your index may be in an inconsistent
> state.
> > > An
> > > existing
> > > > commit.lock indicates such an inconsistent state. Therefore,
> every
> > > writer
> > > needs
> > > > a commit.lock while committing, and every reader needs a
> > > commit.lock while
> > > > opening an index.
> > > >
> > > > Christoph
> > > >
> > > >
> > > >
> > >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail:
> lucene-dev-unsubscribe@jakarta.apache.org
> > > > For additional commands, e-mail:
> lucene-dev-help@jakarta.apache.org
> > > >
> > >
> > >
> > >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail:
> lucene-dev-help@jakarta.apache.org
> > >
> > >
> >
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org