Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-Id: <C8AFD22D-5FC7-46F9-A0A7-A20ED81215FA@mikemccandless.com>
From: Michael McCandless <lucene@mikemccandless.com>
To: java-user@lucene.apache.org
In-Reply-To: <21938848.post@talk.nabble.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: Memory Eaten up by TermInfo Instances in Lucene 2.4
Date: Tue, 10 Feb 2009 15:16:48 -0500
References: <21913262.post@talk.nabble.com>
 <FABB6596-4F91-4A54-A45E-B7A357C1EBC4@mikemccandless.com>
 <21932951.post@talk.nabble.com>
 <2F672EDE-CADA-4793-B9AC-331398A31696@mikemccandless.com>
 <21938848.post@talk.nabble.com>


chanchitodata wrote:

> I actually dont hit OOM, The memory gets 100% full and the JVM hangs.

Is it GC'ing during this hang?  Can you try reducing your heap size  
down alot and see if the GC runs faster?  (Or, if you can provoke an  
OOM).

How large is your heap now?

> Independently what type of GC alogorithm I use. Have tried all sorts  
> of JVM
> GC flags.
> Profiling the application with YourKit I can see that the TermInfo  
> instances
> does not get freed up when the GC is done.

Well... first off, the 100K TermInfos are still referenced and won't  
be freed until
you close the reader.  Likewise, any entries still in the caches are  
still referenced.

How many TermInfo instances do you see still alive before vs after GC?

> The application starts with around 100.000 instances of TermInfo and  
> then
> its starts to accumulate around 1000 and 2000 instances per web  
> request.

This seems high (1-2K per request).  What are your queries like?

How large is the thread pool in your Tomcat instance?

> This is,if I understand you right, normal but shouldn't the allocated
> TermInfo instances be returned after a GC? I never see the instances  
> get
> freed up.

Make sure you look at before/after numbers of TermInfo instances.   
And, reduce your
heap size so that you see GC successfully complete.

> The SegmentReader instances are actually just 3 arrays of  
> SegmentReader
> instances.

But you only have 1 IndexReader open?  (You should only have 1 array  
of SegmentReaders
if so).  Is it possible Compass is holding open some SegmentReaders  
under-the-hood?

Are you (or Compass) ever reopening the reader?

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org