Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com.sg;
  h=X-YMail-OSG:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
  b=APbrJkfBUAXkQL9IZqF8ZKXrXEK5THLbJUeMMvuQQvfR2qcf1SB0cZt7o2Mwy0s1T6Qm4PIpP12YExBM70xKoI32gO7IkVvppQ5q0CjoYgg3KBzz9CNTMR0oMTJh6Typ79OV1rNLlbkdFbaJtyKhzDpk5EDk46gNOWa6DLdn7fU=;
Date: Mon, 6 Oct 2008 23:58:59 +0800 (CST)
From: Edwin Lee <edwin11_1979@yahoo.com.sg>
Subject: RE: Memory eaten up by String, Term and TermInfo?
To: java-user@lucene.apache.org
In-Reply-To: <9263133B61F64864BBA9C7D7E2FD9334@gcheng>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit
Message-ID: <810202.64754.qm@web76011.mail.sg1.yahoo.com>

Hi,

Probably off-topic, but just like to plug a bit on my blog post here:
http://tinyurl.com/4vytcc :p (incidentally, Java GC is one of my favourite
topics) It's not very detailed, but i would like to think it's a good place to
start reading...

Just like to point out a couple of things:

1. If you have a long-running application with quite heavy use of caching, you
should examine your GC logs to see if you are violating the young generation
guarantee, and set NewSize and MaxNewSize if necessary.

2. Regarding GC strategies (e.g. UseConcMarkSweepGC), it really depends on
several factors, such as system specs, what kind of application that you have
(e.g. online transaction or batch processing), and even what the users'
expectations are (high throughput i.e. achieve x number of transactions in a
fixed time period, or low latency i.e. each transaction takes less than x
seconds).

Just some thoughts...


Regards,
Edwin


--- Peter Cheng <petercheng456@gmail.com> wrote:

> Oh yes, I forget to mention that MaxPermSize is very useful, and may be
> another key that solves my problem. I haven't tried UseConcMarkSweepGC and
> the other two parameters, and I will try them instead of my own GC thread to
> see whether the problem can also be solved.
> 
> Thanks Brian!
> 
> Regards,
> Gong
> 
> > -----Original Message-----
> > From: Beard, Brian [mailto:Brian.Beard@mybir.com] 
> > Sent: Monday, October 06, 2008 8:48 PM
> > To: java-user@lucene.apache.org
> > Subject: RE: Memory eaten up by String, Term and TermInfo?
> > 
> > I played around with GC quite a bit in our app and found the following
> > java settings to help a lot (Used with jboss, but should be 
> > good for any
> > jvm).
> > 
> > set JAVA_OPTS=%JAVA_OPTS% -XX:MaxPermSize=512M -XX:+UseConcMarkSweepGC
> > -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled
> > 
> > While these settings did help when the GC starts to get taxed for
> > keeping throughput better, the biggest thing was getting the heap size
> > big enough. If the heap's big enough, then everything seems 
> > to work well
> > regardless of the GC algorithm selected.
> > 
> > If the heap size and algorithm are correct, then you shouldn't have to
> > force GC. Usually the forced GC calls will trigger the total GC which
> > has a long delay and slows down responsiveness to the app.
> > 
> > I found jstat fairly helpful in monitoring all of this. You can see if
> > the following article helps at all.
> > 
> > http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
> > 
> > -----Original Message-----
> > From: Peter Cheng [mailto:petercheng456@gmail.com] 
> > Sent: Sunday, October 05, 2008 7:55 AM
> > To: java-user@lucene.apache.org
> > Subject: RE: Memory eaten up by String, Term and TermInfo?
> > 
> > I have confirmed that the OutOfMemoryError is not Lucene's 
> > problem. It's
> > just because JVM failed to perform GC when necessary, and I don't know
> > why.
> > To fix this, I started another thread to call GC every six hours, and
> > problems got solved.
> > 
> > Thank you all.
> > 
> > Gong
> > 
> > > -----Original Message-----
> > > From: Michael McCandless [mailto:lucene@mikemccandless.com] 
> > > Sent: Sunday, September 14, 2008 10:28 PM
> > > To: java-user@lucene.apache.org
> > > Subject: Re: Memory eaten up by String, Term and TermInfo?
> > > 
> > > 
> > > Small correction: it was checked in this morning (at least, on the  
> > > East Coast of the US).
> > > 
> > > So you need to either build your own JAR using Lucene's trunk, or,  
> > > wait for tonite's build to run and then download the build 
> > artifacts  
> > > from here:
> > > 
> > >      http://hudson.zones.apache.org/hudson/job/Lucene-trunk
> > > 
> > > If possible, please report back if this fixed your OutOfMemoryError.
> > > 
> > > 2.4 will include this fix.
> > > 
> > > Mike
> > > 
> > > Chris Lu wrote:
> > > 
> > > > Can you try to update to the latest Lucene svn version, like  
> > > > yesterday?
> > > > LUCENE-1383 was checked in yesterday. This patch is 
> > > addressing a leak
> > > > problem particular to J2EE applications.
> > > >
> > > > -- 
> > > > Chris Lu
> > > > -------------------------
> > > > Instant Scalable Full-Text Search On Any Database/Application
> > > > site: http://www.dbsight.net
> > > > demo: http://search.dbsight.com
> > > > Lucene Database Search in 3 minutes:
> > > > 
> > > http://wiki.dbsight.com/index.php?title=Create_Lucene_Database
> > > _Search_in_3_minutes
> > > > DBSight customer, a shopping comparison site, (anonymous per  
> > > > request) got
> > > > 2.6 Million Euro funding!
> > > >
> > > > On Sun, Sep 14, 2008 at 6:58 AM, Peter Cheng  
> > > > <petercheng456@gmail.com>wrote:
> > > >
> > > >> Hi the community,
> > > >>
> > > >> In a Tomcat application (a keyword-based search engine), I use  
> > > >> Lucene 2.3.2
> > > >> to index 60 million documents, but each document is 
> > small-sized.  
> > > >> The total
> > > >> index size is about 60GB.
> > > >> After a successful running for a week, Tomcat was down due to  
> > > >> OutOfMemory.
> > > >> Then I restarted Tomcat, and after three days, I used jmap 
> > > and jhat  
> > > >> to find
> > > >> out what had eaten up the memory. I found millions of 
> > > instances of  
> > > >> String,
> > > >> Term, and TermInfo. Why?
> > > >>
> > > >> In my application, I use a single IndexSearcher object, 
> > which is  
> > > >> shared by
> > > >> all the requests. It is opened initially, and will never 
> > be closed.
> > > >>
> > > >> What could have eaten up the memory? What is referring to 
> > > millions of
> > > >> instances of Term and TermInfo?
> > > >>
> > > >> I can provide any snippets of codes if necessary.
> > > >> Thank you so much!
> > > >>
> > > >> Gong Cheng
> > > >>
> > > >>
> > > >> 
> > > 
> > ---------------------------------------------------------------------
> > > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > >> For additional commands, e-mail: java-user-help@lucene.apache.org
> > > >>
> > > >>
> > > 
> > > 
> > > 
> > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > > 
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> > 
> > 
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> > 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 


      Try cool new emoticons, skins, plus more space for friends. 
Download Yahoo! Messenger Singapore now!
http://sg.messenger.yahoo.com


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org