lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie <ja...@stimulussoft.com>
Subject Re: Lucene 3.0 Search Performance Stats
Date Sat, 20 Mar 2010 07:58:50 GMT
Hi Monique

Sure. As soon as I get access to the server again, I'll get the mem 
stats for you. I will say that Lucene was consuming a large amount of 
memory before we moved over to using Numerics. The reason for this is 
that we were encoding dates as strings. Our date time strings were 
unique, so as the number of records exploded, so too did the number of 
terms in the index. As I understand it (and I am no Lucene expert), 
Lucene's sorting mechanisms need to load up all the terms in the index 
in memory during a sort. Thus, if you execute a sorted search, Lucene's 
memory consumption goes through the roof. Using Numerics avoids this 
problem.

There are lot of other strategies we used to reduce memory consumption. 
Like, making sure that you are caching Searchers and IndexReaders, etc.

Regards,

Jamie



On 2010/03/19 07:04 PM, Monique Monteiro wrote:
> Hi Jamie,
>
>    could you please tell us how much memory does your application consume
> with Lucene? I'm asking it because we are having memory consumption problems
> with a 32GB index and 1.5GB od RAM allocated to our web application. At the
> momento, we use textual search.
>   Thanks in advance,
> Monique
> On Fri, Mar 19, 2010 at 8:49 AM, Jamie<jamie@stimulussoft.com>  wrote:
>
>    
>> Hi Guys
>>
>> I just wanted to congratulate the Lucene guys for a fine job on 3.0!!
>>
>> Since we switched our indexes to using integer based range queries based on
>> Date (YYMMHHSS), search speed is lightening fast and memory consumption has
>> dropped considerably!
>>
>> Some stats:
>>
>> Indexed Docs: 7.2M emails
>> Index Size: 24 GB (non optimized)
>> Search Speed: 0.06 - 0.09 seconds (with sort YYMMHHSS date)
>>
>> Index stored on 4 SAS HDD hitachi RAID 10
>> 16G RAM
>> 2x Xeon 4 core 2.4Gz
>> OS FreeBSD 7.2
>> Filesystem UFS2 gjournal
>>
>> I believe we are using all search performance recommendations now.
>>
>> Good job!
>>
>> Jamie
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>>      
>
>    


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message