lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: [Suggestions Required] 110 Concurrency users indexing on Lucene dont finish in 200 ms.
Date Tue, 11 Feb 2014 10:45:11 GMT
Lucene's indexing is very concurrent, especially in 4.x where we've
added concurrent flushing:
http://blog.mikemccandless.com/2011/05/265-indexing-speedup-with-lucenes.html

Are you sure the bottleneck is in Lucene and not e.g. the server-side
handling of suddenly receiving 110 X 10,50,100 documents?  E.g.
perhaps most of your requests are queueing up, if the server only
allows N requests at once.  If you make a standalone Lucene-only test
indexing the same sudden surge of documents, how long does it take?

How many threads actually interact with Lucene?

Mike McCandless

http://blog.mikemccandless.com


On Tue, Feb 11, 2014 at 12:25 AM, Umashanker, Srividhya
<srividhya.umashanker@hp.com> wrote:
> Group -
>
>
>
> We have an Indexing and Searching Service (using Lucene 4.0) implemented over REST  as
part of our framework, which all the related modules will use to publish data  that make it
available for the UI.
>
> Moreover, every rest call that our service receives has a proxy timeout limit of 200
ms.
>
> Initially we started concurrency tests with Indexing.   Recently, we started doing some
concurrency tests on our service to handle  min 110 users, who will be publishing json data
ranging from 10, 50 100 documents by one user. The document is not huge like what u are using
for your performance tests. Mostly we call updateDocument() as we do not know whether the
document is already indexed or not
>
>
>
> 110 users/threads with a ramp up time of 0 seconds seems to fail randomly because some
of the threads/REST call do not complete within 200 ms.
>
>
>
> We tried the following.
>
> 1. maxBufferSize of 500 MB.
>
> 2. DWPT count to 10.
>
> 3. Tried updateDocument in separate thread, as said in one of your suggestions in the
internet. Doesn't seem to help.
>
> 4. TiredMergePolicy has been set. Unfortunately this does not help as the maxBufferSize
is not hit while we index, and merge does not happen yet.
>
> 5. We are yet to try with NMapDirectory.
>
>
>
> Are there any other suggestions/experiences that you can share with us.  Are there any
benchmark tests that you have done on concurrency?
>
>
>
> Please help us.
>
>
>
> -Vidhya
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message