Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.co.in;
  h=Received:X-YMail-OSG:X-Yahoo-Newman-Property:Message-ID:From:To:References:Subject:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:X-Priority:X-MSMail-Priority:X-Mailer:X-MimeOLE;
  b=r59OkAIhCJgw7sOb8QSBmJWzf09FLWjoxs2t0MAgDW5Vd3oRfXq9ZqKWdsdlaeb4K1IEwj7FDKqo3dN3/nFoTDE5wtoCY0yVuFHFT0rODWs0TVYOYSSHEw9SjaLb5pEs8EybK/GwbekNnnDxq4RQcHF+2sGMlGB1asKGj6jRB1M=
  ;
Message-ID: <123a01c95541$5449fce0$330bc30a@sv.us.sonicwall.com>
From: "Ganesh" <emailgane@yahoo.co.in>
To: <java-user@lucene.apache.org>
References: <D4A35A3A-5909-4096-9491-C9D839925929@mac.com>
Subject: Re: lucene nicking all my memory
Date: Wed, 3 Dec 2008 17:49:00 +0530
MIME-Version: 1.0
Content-Type: text/plain;
	format=flowed;
	charset="iso-8859-1";
	reply-type=response
Content-Transfer-Encoding: 7bit

You are opening and closing IndexSearcher for every search.  Try by caching 
IndexSearcher and do reopen the IndexReader, when the index gets modified.

In your code below, How did you create IndexSearcher. If it is using 
IndexReader and you need to close that too. This might be the cause of 
memory leak.

Regards
Ganesh


----- Original Message ----- 
From: "Magnus Rundberget" <rundis@mac.com>
To: <java-user@lucene.apache.org>
Sent: Wednesday, December 03, 2008 4:03 PM
Subject: lucene nicking all my memory


> Hi,
>
> We have an application using Tomcat, Spring etc and Lucene 2.4.0.
> Our index is about 100MB (in test) and has about 20 indexed fields.
>
> Performance is pretty good, but we are experiencing a very high usage  of 
> memory when searching.
>
> Looking at JConsole during a somewhat silly scenario (but illustrates  the 
> problem);
> (Allocated 512 MB Min heap space, max 1024)
>
> 0. Initially memory usage is about 70MB
> 1. Search for word "er", heap memory usage goes up by 100-150MB
> 1.1 Wait for 30 seconds... memory usage stays the same (ie no gc)
> 2. Search by word "og", heap memory usage goes up another 50-100MB
> 2.1 See 1.1
>
> ...and so on until it seems to reach the 512 MB limit, and then a  garbage 
> collection is performed
> i.e garbage collection doesn't seem to occur until it "hits the roof"
>
> We believe the scenario is similar in production, were our heap space  is 
> limited to 1.5 GB.
>
>
> Our search is basically as follows
> ----------------------------------------------
> 1. Open an IndexSearcher
> 2. Build a Boolean Query searching across 4 fields (title, summary, 
> content and daterangestring YYYYMMDD)
> 2.1 Sort on title
> 3. Perform search
> 4. Iterate over hits to build a set of custom result objects (pretty 
> small, as we dont include content in these)
> 5. Close searcher
> 6. Return result objects.
>
>
> We have tried various options based on entries on this mailing list;
> a) Cache the IndexSearcher - Same results
> b) Remove sorting - Same result
> c) In point 4 only iterating over a limited amount of hits rather than 
> whole collection - Same result in terms of memory usage, but obviously 
> increased performance
> d) Using RamDirectory vs FSDirectory - Same result only initial heap 
> usage is higher using ramdirectory (in conjuction with cached 
> indexsearcher)
>
>
> Doing some profiling using YourKit shows a huge number of char[],  int[] 
> and string[], and ever increasing number of lucene related  objects.
>
>
>
> Reading through the mailing lists, suspicions are that our problem is 
> related to ThreadLocals and memory not being released. Noticed that  there 
> was a related patch for this in 2.4.0, but it doesn't seem to  help us 
> much.
>
> Any ideas ?
>
> kind regards
> Magnus
>
>
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

Send instant messages to your online friends http://in.messenger.yahoo.com 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org