lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Optimizing NRT search
Date Fri, 26 Apr 2013 10:46:58 GMT
Batching the updates really ought to improve overall throughput.  Have you
tried with even bigger batches (100,1000 docs)?

But, how large is each update?  Are you changing any IndexWriter settings,
e.g. ramBufferSizeMB.

Using threads should help too, at least a separate thread doing indexing
from calling SearcherManager.maybeRefresh (and, separate threads doing
searching).

You can also check out
http://blog.mikemccandless.com/2011/06/lucenes-near-real-time-search-is-fast.htmlwhere
I go into some detail on speeding up indexing rate and refresh speed
with near-real-time ...

Mike McCandless

http://blog.mikemccandless.com


On Thu, Apr 25, 2013 at 11:10 PM, Aleksey <bittercold@gmail.com> wrote:

> Hey guys,
>
> I'm new to Lucene and I was trying to estimate how fast I can make updates
> to the index and reopen it. The behavior I'm seeing seems odd.
> I'm using Lucene4.2 and SearchManager instance that takes an index writer.
>
> I make a loop where I update 1 document, then call maybeRefresh and acquire
> new searcher and make a search to verify that the update is there. On my
> laptop this does about 100 iterations per second.
>
> Then I run another loop but make 10 updates before reopening the index, and
> this only does 10 iterations per second, proportionally less. I was
> expecting that if I batch the updates, I can get higher overall throughput,
> but that does not seem to be the case. The size of the index I'm updating
> doesn't make a difference either, I tried 3K and 100K document sets, 1-2K
> each doc, but both produce same update speed (though I'm not calling commit
> in these instances).
>
> Can anyone point me into the right direction to investigate this or hint
> how to maximize write throughput to the index, while still <.5 second
> delays in seeing the updates.
>
> Thank you in advance,
>
> Aleksey
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message