lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <luc...@mikemccandless.com>
Subject RE: Threading Indexing Processes : Can we write concurrently to Index?
Date Wed, 31 Oct 2007 10:20:17 GMT

Just to clarify here: yes, you really should have a single JVM with a
single instance of IndexWriter, but use multiple threads calling
IndexWriter.addDocument.

Under the hood, IndexWriter can make use of alot of concurrency, so
you should see a substantial gain in indexing throughput if you use
multiple threads.  Thread scale up has been tested (as part of the
speedups for indexing) with the upcoming 2.3 release:

  http://issues.apache.org/jira/browse/LUCENE-843#action_12502793

No additional locking is required on your part: addDocument,
internally, takes care of being synchronized in the right places.
Just make sure all your threads stop adding documents when you finally
call IndexWriter.close().

But please note: this is only the case if the bottleneck in your app
is really Lucene's indexing.  EG, if you are pulling docs from some
external repository, and that repository is the bottleneck, then you
shouldn't expect any real gains by using multiple threads.

Mike

"askxuefeng" <askxuefeng@gmail.com> wrote:
> You can use a queue to let IndexWriter do its work asychronize 
> 
> -----Original Message-----
> From: Michael Prichard [mailto:michael_prichard@mac.com] 
> Sent: Tuesday, October 30, 2007 9:10 PM
> To: java-user@lucene.apache.org
> Subject: Re: Threading Indexing Processes : Can we write concurrently to
> Index?
> 
> So I should create a singe IndexWriter but allow multiple threads to use
> it?
> 
> 
> On Oct 30, 2007, at 12:49 AM, Karl Wettin wrote:
> 
> >
> > 30 okt 2007 kl. 05.08 skrev Michael Prichard:
> >
> >> If I multithread an indexing process can two or more processes write 
> >> to the same index?
> >
> > Multiple threads can add documents to the same writer. Multiple 
> > threads can usually not write using each a writer.
> >
> >
> > --
> > karl
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message