lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pete Lewis" <p...@uptima.co.uk>
Subject Re: Lock handling and Lucene 1.9 / 2.0
Date Wed, 15 Sep 2004 20:22:14 GMT
Hi Otis

Our configuration is multi-JVM and single shared disk (4 * servers, 1 *
SAN).  Even having multiple dedicated IndexReaders would still cause us
iisues because of the shared disk access for the commit.lock file.....

Why are we being exclusive read locked for just a search?  I know we are not
using Lucene in the 'standard way' but feel that we've laid things out in
the best way for our environment, which is driven by issues other than just
search.

Yep, library == Lucene index.  Sorry, also working with different search
engine in the Terabyte range and they call their indexes 'libraries' and I
forget to make the switch sometimes.

Cheers

Pete Lewis

----- Original Message ----- 
From: "Otis Gospodnetic" <otis_gospodnetic@yahoo.com>
To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
Sent: Wednesday, September 15, 2004 12:17 PM
Subject: Re: Lock handling and Lucene 1.9 / 2.0


> Pete,
>
> You can use the same IndexReader by multiple threads.  It sounds like
> you are not aware of this (I think this is a FAQ actually... on jGuru).
> So, single IndexReader used by multiple threads == OK && does not
> require commit.lock on ever 'read'.  Just one commit.lock when
> IndexReader is opened, or when you use IndexReader to delete a
> Document.
>
> When you say 'library', you are really referring to a Lucene index,
> correct?
>
> Otis
>
> --- Pete Lewis <pete@uptima.co.uk> wrote:
>
> > Hi Doug
> >
> > Weblogic creates a pool of objects for us which are re-initialised
> > each time
> > the constructor is called.  Its when we grab an IndexReader out of
> > the pool
> > that we have the creation of the cache, which is where the spin
> > originates.
> >
> > Been thinking about your suggestion of a Hashtable (based on library
> > name)
> > for the storage of IndexReaders, but then we'd get a bottleneck of
> > access -
> > having a single reader per library means only a single process can
> > access
> > the library (for reading) at once, and this would create a potential
> > bottleneck across our servers.  Another way might be to create a pool
> > of
> > IndexReaders and allocate them on demand, ie 10 IndexReaders per
> > library.
> > This would allow for 10 synchronous searches with no commit lock
> > spin, but
> > would be a pain to code.
> >
> > Probably would be quickest to create a system property that will
> > enable us
> > to turn on/off the commit lock around the FSDirectory cache creation,
> > so
> > we'd have them off when we get an IndexReader for just a read but
> > have the
> > locks on at other times - don't want to disable all locks as our
> > libraries
> > are dynamic and not static.
> >
> > Sorry the constructive criticism was off-the-wall but it had made my
> > head
> > hurt getting to the bottom of where our waits on spin locks had come
> > from
> > ;-)
> >
> > Cheers
> >
> > Pete Lewis
> >
> > ----- Original Message ----- 
> > From: "Doug Cutting" <cutting@apache.org>
> > To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
> > Sent: Tuesday, September 14, 2004 10:12 PM
> > Subject: Re: Lock handling and Lucene 1.9 / 2.0
> >
> >
> > > Pete Lewis wrote:
> > > > The only way to continually use the same IndexReader would be to
> > use a
> > > > stateful session bean (frowned upon by J2EE Container writers)
> > >
> > > Can one implement DB connection pooling in J2EE?  This is
> > analogous.
> > > One may keep a pool of IndexReaders that are reused by subsequent
> > > queries.  One difference is that the cache need only contain a
> > single
> > > IndexReader per index, rather than a DB connection pool, which
> > typically
> > > keeps multiple connections per DB.  Also, at checkout, the cache
> > code
> > > should check whether a newer version of the index is available,
> > and, if
> > > it is, update the cache.
> > >
> > > If there are lots of different indexes, more than you'd like to
> > keep
> > > open at once, then an LRU cache might work well, implemented e.g.
> > with
> > > LinkedHashMap.  Such a cache might be a useful contribution to
> > Lucene.
> > >
> > > > I thought that it might be a good candidate for Lucene 2 as the
> > FSDirectory
> > > > code is horrible and non-J2EE compliant.
> > >
> > > Your constructive criticism is greatly appreciated!
> > >
> > > Have a nice day,
> > >
> > > Doug
> > >
> > >
> > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message