lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <jha...@fibertel.com.ar>
Subject Re: QueryParser not thread-safe
Date Wed, 24 Aug 2005 15:10:51 GMT
Thanks for all the answers! 

Now, while I agree that simple is better, I think QueryParser should 
be made thread-safe through the use of a wrapper class (along the 
line of what Luc suggested) or by using something other than javacc, 
so this is no longer an issue in future lucene versions.

I caught the error through a load test on a system with low memory 
and slow processor, which exacerbates the concurrency issues. 
It was a hard to catch bug: even when it showed up the query results 
still kinda made sense, because missing or duplicate search terms 
don't produce complete garbage.

I'm a beginner, so I may be missing something big, but if not, many 
lucene installations out there may be having this problem without 
knowing it.


> -----Mensaje original-----
> De: Chris Lamprecht [mailto:clamprecht@gmail.com]
> Enviado el: MiƩrcoles, 24 de Agosto de 2005 07:20
> Para: java-user@lucene.apache.org
> Asunto: Re: QueryParser not thread-safe
> 
> 
> I would just create a new QueryParser for each query.  Allocating
> short-lived objects is just about free in java, and the time spent
> performing the actual search will by far dominate any time spent
> constructing QueryParser objects.
> 
> On 8/24/05, Vanlerberghe, Luc <Luc.Vanlerberghe@bvdep.com> wrote:
> > Thanks for pointing that out!
> > I checked the source and QueryParser is indeed not thread-safe (the
> > presence of local variables like jj_lastpos that are used *during* the
> > parsing makes this obvious)
> > 
> > Perhaps it should be explicitly mentioned in the javadoc.
> > 
> > The solution I'll probably go for is using ThreadLocal variables
> > (available since jdk 1.2, but there was a problem in the sun jvm with
> > deallocation when threads stop until jdk 1.4 or so)
> > 
> > an example (not compiled, nor tested):
> > 
> > public class MyQueryFactory {
> >    public Query createQuery(String expression) {
> >        return threadLocalQueryParser.get().parse(expression);
> >    }
> > 
> >    private ThreadLocal<QueryParser> threadLocalQueryParser=new
> > ThreadLocal<QueryParser>() {
> >        protected QueryParser initialValue() {
> >            return new QueryParser("defaultField", new MyAnalyzer());
> >        }
> >    };
> > }
> > 
> > (I used jdk 1.5 generics because the code is clearer, remove the
> > <QueryParser> and put a few casts where needed if you want this in 1.4
> > or lower)
> > 
> > initialValue() will be only be called once for every thread that calls
> > threadLocalQueryParser.get().
> > 
> > See the javadoc for java.lang.ThreadLocal
> > 
> > Luc
> > 
> > 
> > -----Original Message-----
> > From: jian chen [mailto:chenjian1227@gmail.com]
> > Sent: dinsdag 23 augustus 2005 21:38
> > To: java-user@lucene.apache.org
> > Subject: Re: QueryParser not thread-safe
> > 
> > Right. My philosophy is that, make it work, then, make it better.
> > 
> > Don't waste time on something that you are not sure if it would cause
> > performance problem.
> > 
> > Jian
> > 
> > On 8/23/05, Paul Elschot <paul.elschot@xs4all.nl> wrote:
> > > On Tuesday 23 August 2005 19:01, Miles Barr wrote:
> > > > On Tue, 2005-08-23 at 13:47 -0300, jhandl@fibertel.com.ar wrote:
> > > > > Hi! I've been having problems with lucene's QueryParser, apparently
it is not thread-safe.
> > > > >
> > > > > That means I can't parse queries in threads where the queryparser
object
> > > > > is created once and reused for each query. If I do, the resulting
queries may
> > > > > have all kinds of weird problems, for example missing terms, duplicate
terms, etc.
> > > > >
> > > > > I don't like the idea of creating a new queryparser for each query,
so I
> > > > > am looking for alternatives. Any ideas?
> > > >
> > > > Using a non-threadsafe object in a threaded environment is fairly
> > > > standard in Java, just wrap it in a synchronized block.
> > > >
> > > > If you don't want all threads waiting on one query parser, create a pool
> > > > of them.
> > >
> > > Or use one parser per thread.
> > >
> > > Regards,
> > > Paul Elschot



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message