lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Magnus Rundberget <run...@mac.com>
Subject Re: lucene nicking all my memory
Date Wed, 03 Dec 2008 12:34:31 GMT
Cheers,

In my scenario. I've made sure that the index does not get modified  
(so reopen shouldnt be necessary ?).

I've tried scenario with both caching and not caching indexsearcher  
(and hereby the indexreader it creates in its constructor).
When not caching Ive made sure to close the indexsearcher (and its  
close method should close the reader it contains).
Also tried with creating an reader explicit, and opening/closing it.

All awail.

Btw creation of index reader is done as follows in the scenario below;
IndexSearcher searcher = new IndexSearcher("PathTOIndexDirectory"); /

.. and closing it
searcher.close();

so I dont believe the solution is as simple as either;
1. Cache the IndexReader (tried and tested with no luck)
2. Open/close the indexReader (implicitly through IndexSearcher or  
explicitly) for each search

something else is fishy :-)

regards
Magnus







On 3. des.. 2008, at 13.19, Ganesh wrote:

> You are opening and closing IndexSearcher for every search.  Try by  
> caching IndexSearcher and do reopen the IndexReader, when the index  
> gets modified.
>
> In your code below, How did you create IndexSearcher. If it is using  
> IndexReader and you need to close that too. This might be the cause  
> of memory leak.
>
> Regards
> Ganesh
>
>
> ----- Original Message ----- From: "Magnus Rundberget"  
> <rundis@mac.com>
> To: <java-user@lucene.apache.org>
> Sent: Wednesday, December 03, 2008 4:03 PM
> Subject: lucene nicking all my memory
>
>
>> Hi,
>>
>> We have an application using Tomcat, Spring etc and Lucene 2.4.0.
>> Our index is about 100MB (in test) and has about 20 indexed fields.
>>
>> Performance is pretty good, but we are experiencing a very high  
>> usage  of memory when searching.
>>
>> Looking at JConsole during a somewhat silly scenario (but  
>> illustrates  the problem);
>> (Allocated 512 MB Min heap space, max 1024)
>>
>> 0. Initially memory usage is about 70MB
>> 1. Search for word "er", heap memory usage goes up by 100-150MB
>> 1.1 Wait for 30 seconds... memory usage stays the same (ie no gc)
>> 2. Search by word "og", heap memory usage goes up another 50-100MB
>> 2.1 See 1.1
>>
>> ...and so on until it seems to reach the 512 MB limit, and then a   
>> garbage collection is performed
>> i.e garbage collection doesn't seem to occur until it "hits the roof"
>>
>> We believe the scenario is similar in production, were our heap  
>> space  is limited to 1.5 GB.
>>
>>
>> Our search is basically as follows
>> ----------------------------------------------
>> 1. Open an IndexSearcher
>> 2. Build a Boolean Query searching across 4 fields (title, summary,  
>> content and daterangestring YYYYMMDD)
>> 2.1 Sort on title
>> 3. Perform search
>> 4. Iterate over hits to build a set of custom result objects  
>> (pretty small, as we dont include content in these)
>> 5. Close searcher
>> 6. Return result objects.
>>
>>
>> We have tried various options based on entries on this mailing list;
>> a) Cache the IndexSearcher - Same results
>> b) Remove sorting - Same result
>> c) In point 4 only iterating over a limited amount of hits rather  
>> than whole collection - Same result in terms of memory usage, but  
>> obviously increased performance
>> d) Using RamDirectory vs FSDirectory - Same result only initial  
>> heap usage is higher using ramdirectory (in conjuction with cached  
>> indexsearcher)
>>
>>
>> Doing some profiling using YourKit shows a huge number of char[],   
>> int[] and string[], and ever increasing number of lucene related   
>> objects.
>>
>>
>>
>> Reading through the mailing lists, suspicions are that our problem  
>> is related to ThreadLocals and memory not being released. Noticed  
>> that  there was a related patch for this in 2.4.0, but it doesn't  
>> seem to  help us much.
>>
>> Any ideas ?
>>
>> kind regards
>> Magnus
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> Send instant messages to your online friends http://in.messenger.yahoo.com
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message