lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eks dev <>
Subject Re: Post mortem kudos for (LUCENE-843) :)
Date Fri, 13 Jul 2007 08:33:39 GMT
Hi Mike, 
> Was 24M (and not more) clearly the fastest performance?

No, this is kind of optimum. Throwing more memory up to 32M makes things slightly faster at
slow rate, having maximum at 32.  After that things start getting slower   (slowly)

We are not yet completely done with tuning, especially  with two tips  you mentioned in this
Fields are already reused, but

1. Reusing Document, this is one new Vector() in there  (and at these speeds, something like
this  makes difference!!!) 
in Document  List fields = new Vector(); (by the way, must this be synchronized  Vector? Why
not ArrayList? Any difference from it)

2. Reusing Field, excuse my ignorance, but how I can do it? with Document is easy with 
luceneDocument.removeFields(name) //Wouldn't be better to have luceneDocument.removeAllFields()

3. "LUCENE-845" Whoops, I totally overlooked this one! And I am sure my maxBufferedDocs is
well under what fits in 24Mb?!?  Any good tip on how to determine good number: count added
docs and see how far this number goes before flush() triggers (how I detect when flush by
ram gets triggered?) and than add 10% to this number...

----- Original Message ----
From: Michael McCandless <>
Sent: Thursday, 12 July, 2007 9:22:48 PM
Subject: Re: Post mortem kudos for (LUCENE-843) :)

Thank you for the compliments, and thank you for being such early
adopter testers!  I'm very glad you didn't hit any issues :)

> before LUCENE-843 indexing speed was 5-6k records per second (and I
> believed this was already as fast as it gets)
> after (trunk version yesterday) 60-65k documents per second! All
> (exhaustive!) tests pass on this index.

Wow, 10X speedup is even faster than my fastest results!

> autocommit = false, 24M RAMBuffer, using char[] instead of String
> for Token (this was the reason we separated Analysis in two phases,
> leaving for Lucene Analyzer only simple whitespace tokenization)

Looks like you're doing everything right to get fastest performance.

You can also re-use the Document & Field instances, and also the Token
instance in your analyzer and that should help more.

Was 24M (and not more) clearly the fastest performance?

Also note that you must workaround LUCENE-845 (still open):;jsessionid=E110C767DA8EFEC5B7D39D00EEF1EB74

You should set your maxBufferedDocs to something "close to but always
above" how many docs actually get flushed after 24 MB RAM is full,
else you could spend way too much time merging.  I'm working on
LUCENE-845 now but not yet sure when it will be resolved...


To unsubscribe, e-mail:
For additional commands, e-mail:

What kind of emailer are you? Find out today - get a free analysis of your email personality.
Take the quiz at the Yahoo! Mail Championship.* 

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message