lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eks dev <eks...@yahoo.co.uk>
Subject Re: Post mortem kudos for (LUCENE-843) :)
Date Fri, 13 Jul 2007 14:27:29 GMT
>Interesting.  This matches the experience Doron had where adding more
>RAM actually slowed things down a bit (posted to
>LUCENE-843).

I know this intrigues you, so our fresh experience:

The bigger the RAM Buffer the faster indexing, this holds until you hit some limit that starts
irritating gc(). But this limit  is somehow "natural" and is given by the environment (available
RAM, competing OS File cache and who knows what else)... basically , we concluded  after testing
the more memory, more speed is to expect (this is kind of ideal scaling, one  proof more of
the algorithmic strength). 

This test showed 110k Docs/second at 32Mb (what we found to be optimal for our needs, as it
slowly speeds-up after that to 123k Docs/sec at 256Mb)

I suspect this phenomena on our last test and what Doron mentioned was due to the wrong maxBufferedDocs.
Have no other explanation

Basically, we achieved almost 20 X speed-up by just having LUCEN-843 and your valuable comments
on how to utilize this nice machine called Lucene.  








      ___________________________________________________________ 
Yahoo! Mail is the world's favourite email. Don't settle for less, sign up for
your free account today http://uk.rd.yahoo.com/evt=44106/*http://uk.docs.yahoo.com/mail/winter07.html



---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message