lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: [Suggestions Required] 110 Concurrency users indexing on Lucene dont finish in 200 ms.
Date Thu, 13 Feb 2014 11:32:20 GMT
For better performance, you should not send 100 threads to
IndexWriter, but rather a number of threads in proportion to how many
CPUs the machine has.  E.g. if your CPU has 8 cores then use at most
12 (=8 * 1.5) indexing threads. It's fine to have 100 client threads
sending documents, but drop these documents into a queue and have the
12 indexing threads pull from there.

If you will have more than 8 threads in IndexWriter at once, then you
should call IndexWriterConfig.setMaxThreadStates to increase the
default (8).

Also, your benchmark does not allow for JVM warming, so you are
measuring e.g. hotspot compilation time.  It's better to make a long
running test and then measure the indexing throughput at steady state,
once the JVM is warmed.

In Lucene's nightly benchmark
(https://people.apache.org/~mikemccand/lucenebench/indexing.html ), we
index ~ 1 KB sized docs around 44.9 Kdocs/sec, or ~ 145 GB/hour, but
that's a very different test than what you are running (e.g., it uses
.addDocument not the more costly .updateDocument)...

Mike McCandless

http://blog.mikemccandless.com


On Thu, Feb 13, 2014 at 5:08 AM, sree <mail2sreedeep@gmail.com> wrote:
> Thanks for your reply. We are using 100 threads and each indexes 100
> documents. Now we created a standalone project which uses lucene to index
> 100 documents for 100 theads concurrently and we can see that each thread
> uses an average of more than 1 sec.
>
> lucene-group.zip
> <http://lucene.472066.n3.nabble.com/file/n4117133/lucene-group.zip>
> <http://lucene.472066.n3.nabble.com/file/n4117133/lucene_profiler.jpg>
>
>
> Please find attached source files, excel sheet and profiler image for more
> information.
>
> thanks
> Sreedeep
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Suggestions-Required-110-Concurrency-users-indexing-on-Lucene-dont-finish-in-200-ms-tp4116625p4117133.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message