lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Indexing Speed: 2.3 vs 2.2 (real world numbers)
Date Mon, 04 Feb 2008 10:51:39 GMT

Even pre-2.3, you should have seen gains by adding threads, if indeed  
your hardware has good concurrency.

And definitely with the changes in 2.3, you should see gains by  
adding threads.

Note that as you add threads, the "sweet spot" for RAM buffer size  
increases.  Ie, make the RAM buffer bigger as you add more threads.

I think the only major thing that's single-threaded is flushing a new  
segment to disk.  Only one thread can do that, and while that thread  
is doing so, other threads must wait.

Mike

Jake Mannix wrote:

> ------=_Part_3862_23986701.1202102642086
> Content-Type: text/plain; charset=ISO-8859-1
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
>
> The test in which we got the 11X speedup?  That was single  
> threaded.  I
> haven't yet found a way to make multithreaded (shared IndexWriter)  
> indexing
> perform with any better speed than singlethreaded, so that code is not
> enabled in our tests.  Do you think that 2.3 would better take  
> advantage of
> multiple threads / cores?  If so, I could rerun it again  
> multithreaded and
> see if that's even better...
>
>   -jake
>
> On Feb 3, 2008 9:02 PM, ajay_garg  
> <garg.jeena.isi.ka.naam.hai@gmail.com>
> wrote:
>
>>
>> Hi Jake.
>>
>> Was the test conducted with a single indexing thread, or multiple  
>> ones ?
>>
>>
>> Jake Mannix wrote:
>>>
>>> Hello all,
>>>   I know you lucene devs did a lot of work on indexing  
>>> performance in
>> 2.3,
>>> and I just tested it out last thursday, so I thought I'd let you  
>>> know
>> how
>>> it
>>> fared:
>>>
>>>   On a 2.17 million document index, a recent test gave indexing  
>>> time to
>>> be:
>>>
>>>     * lucene 2.2: 4.83 hours
>>>     * lucene 2.3: 26 minutes
>>>
>>>   About a factor of 11 speedup.  Holy smokes!  Great work folks.
>>>
>>>
>>>   -jake
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Indexing-Speed%3A-2.3-vs-2.2-%28real-world- 
>> numbers%29-tp15257512p15262216.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message