lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: Lucene 3.0 Search Performance Stats
Date Sat, 20 Mar 2010 23:27:48 GMT
Hi Jamie,

thanks for reporting back the numbers about your usage of NumericField and NumericRangeQuery!
I am glad to hear about it.

> Sure. As soon as I get access to the server again, I'll get the mem
> stats for you. I will say that Lucene was consuming a large amount of
> memory before we moved over to using Numerics. The reason for this is
> that we were encoding dates as strings. Our date time strings were
> unique, so as the number of records exploded, so too did the number of
> terms in the index. As I understand it (and I am no Lucene expert),
> Lucene's sorting mechanisms need to load up all the terms in the index
> in memory during a sort. Thus, if you execute a sorted search, Lucene's
> memory consumption goes through the roof. Using Numerics avoids this
> problem.

Thats true, as you only need to store an integer per document in the cache (and not a String).
Also performance for FieldCache warmup and sorting is higher.

> There are lot of other strategies we used to reduce memory consumption.
> Like, making sure that you are caching Searchers and IndexReaders, etc.
> 
> Regards,
> 
> Jamie
> 
> 
> 
> On 2010/03/19 07:04 PM, Monique Monteiro wrote:
> > Hi Jamie,
> >
> >    could you please tell us how much memory does your application
> consume
> > with Lucene? I'm asking it because we are having memory consumption
> problems
> > with a 32GB index and 1.5GB od RAM allocated to our web application.
> At the
> > momento, we use textual search.
> >   Thanks in advance,
> > Monique
> > On Fri, Mar 19, 2010 at 8:49 AM, Jamie<jamie@stimulussoft.com>
> wrote:
> >
> >
> >> Hi Guys
> >>
> >> I just wanted to congratulate the Lucene guys for a fine job on
> 3.0!!
> >>
> >> Since we switched our indexes to using integer based range queries
> based on
> >> Date (YYMMHHSS), search speed is lightening fast and memory
> consumption has
> >> dropped considerably!
> >>
> >> Some stats:
> >>
> >> Indexed Docs: 7.2M emails
> >> Index Size: 24 GB (non optimized)
> >> Search Speed: 0.06 - 0.09 seconds (with sort YYMMHHSS date)
> >>
> >> Index stored on 4 SAS HDD hitachi RAID 10
> >> 16G RAM
> >> 2x Xeon 4 core 2.4Gz
> >> OS FreeBSD 7.2
> >> Filesystem UFS2 gjournal
> >>
> >> I believe we are using all search performance recommendations now.
> >>
> >> Good job!
> >>
> >> Jamie
> >>
> >>
> >>
> >> --------------------------------------------------------------------
> -
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
> >>
> >
> >
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message