lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <luc...@mikemccandless.com>
Subject Re: Post mortem kudos for (LUCENE-843) :)
Date Fri, 13 Jul 2007 14:41:56 GMT

"eks dev" <eksdev@yahoo.co.uk> wrote:
> >Interesting.  This matches the experience Doron had where adding more
> >RAM actually slowed things down a bit (posted to
> >LUCENE-843).
> 
> I know this intrigues you, so our fresh experience:

Yes it does!  Thanks :)

> The bigger the RAM Buffer the faster indexing, this holds until you hit
> some limit that starts irritating gc(). But this limit  is somehow
> "natural" and is given by the environment (available RAM, competing OS
> File cache and who knows what else)... basically , we concluded  after
> testing the more memory, more speed is to expect (this is kind of ideal
> scaling, one  proof more of the algorithmic strength). 

OK, this is a good datapoint.  It sound like one has to test in their
own environment to find the optimal performance / RAM usage tradeoff.

> This test showed 110k Docs/second at 32Mb (what we found to be optimal
> for our needs, as it slowly speeds-up after that to 123k Docs/sec at
> 256Mb)
> 
> I suspect this phenomena on our last test and what Doron mentioned was
> due to the wrong maxBufferedDocs. Have no other explanation
> 
> Basically, we achieved almost 20 X speed-up by just having LUCEN-843 and
> your valuable comments on how to utilize this nice machine called
> Lucene. 

WOW!  That's great :)

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message