lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Toke Eskildsen ...@statsbiblioteket.dk>
Subject Re: Lucene performance issues..
Date Mon, 28 Jul 2008 08:47:24 GMT
On Sun, 2008-07-27 at 21:38 +0100, Mazhar Lateef wrote:
>   * email searching
>         o We are creating very large indexes for emails we are
>           processing, the size is upto +150GB for indexes only (not
>           including data content), this we thought would improve
>           search performance since less indexes to open and read from,
>           however the searching taking upto minutes and sometime never
>           returns results

It this with or without warm-up? How many hits does a query typically
return and what do you do with those hits?

> We have also tried upgrading the lucene version to 2.3 in hope to 
> improve performance but the results were quite the opposite. but from my 
> research on the internet the Lucene version 2.3 is much faster and 
> better so why are we seeing such inconsistency.

I encountered the same problem some time ago. It turns out that you get
lower performance if you're using an index from an older version of
Lucene with a newer version. If you haven't done so already, try
converting the old index to the new format.

Another way to go is Solid State Drives. While not cheap in themselves,
the performance-increase might make a purchase favorable. One of the
fine properties of Lucene on SSD, besides the general increase in speed,
if that less warm-up is required. This means that frequent updates of a
large index without a huge performance-hit is attainable. For a 150GB+
index, you would probably want to go for 256GB of storage and split the
index in 64GB chunks, to avoid running out of storage during merge.

We've put some fragmented notes and observations on the subject at
http://wiki.statsbiblioteket.dk/summa/Hardware - I apologize for not
taking the time to polish it, but an important deadline is looming.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message