lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Boyd <andrew.b...@mindspring.com>
Subject Re: Fastest way to fetch N documents with unique keys within large numbers of indexes..
Date Thu, 09 Jun 2005 08:50:07 GMT
Kevin,
  Those results are awsome.  Could you please give those of us that were following but not
quite understanding everything some pseudo code or some more explaination?

Thanks,

andrew

-----Original Message-----
From: Kevin Burton <burton@rojo.com>
Sent: Jun 7, 2005 7:18 PM
To: java-user@lucene.apache.org
Subject: Re: Fastest way to fetch N documents with unique keys within large numbers of indexes..

Paul Elschot wrote:

>For a large number of indexes, it may be necessary to do this over
>multiple indexes by first getting the doc numbers for all indexes,
>then sorting these per index, then retrieving them
>from all indexes, and repeating the whole thing using terms determined
>from the retrieved docs.
>  
>
Well this was a BIG win.  Just benchmarking it out shows a 10x -> 50x 
performance increase.

Times in milliseconds:

Before:

duration: 1127
duration: 449
duration: 394
duration: 564

After:

duration: 182
duration: 39
duration: 12
duration: 11

The values of 2-4  I'm sure are due to the filesystem buffer cache but I 
can't imagine why they'd be faster in the second round.  It might be 
that Linux is deciding not to buffer the document blocks.

Kevin

-- 

Use Rojo (RSS/Atom aggregator)! - visit http://rojo.com. 
See irc.freenode.net #rojo if you want to chat.

Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html

   Kevin A. Burton, Location - San Francisco, CA
      AIM/YIM - sfburtonator,  Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message