lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Cheng" <petercheng...@gmail.com>
Subject RE: Memory eaten up by String, Term and TermInfo?
Date Mon, 06 Oct 2008 15:11:25 GMT
Oh yes, I forget to mention that MaxPermSize is very useful, and may be
another key that solves my problem. I haven't tried UseConcMarkSweepGC and
the other two parameters, and I will try them instead of my own GC thread to
see whether the problem can also be solved.

Thanks Brian!

Regards,
Gong

> -----Original Message-----
> From: Beard, Brian [mailto:Brian.Beard@mybir.com] 
> Sent: Monday, October 06, 2008 8:48 PM
> To: java-user@lucene.apache.org
> Subject: RE: Memory eaten up by String, Term and TermInfo?
> 
> I played around with GC quite a bit in our app and found the following
> java settings to help a lot (Used with jboss, but should be 
> good for any
> jvm).
> 
> set JAVA_OPTS=%JAVA_OPTS% -XX:MaxPermSize=512M -XX:+UseConcMarkSweepGC
> -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled
> 
> While these settings did help when the GC starts to get taxed for
> keeping throughput better, the biggest thing was getting the heap size
> big enough. If the heap's big enough, then everything seems 
> to work well
> regardless of the GC algorithm selected.
> 
> If the heap size and algorithm are correct, then you shouldn't have to
> force GC. Usually the forced GC calls will trigger the total GC which
> has a long delay and slows down responsiveness to the app.
> 
> I found jstat fairly helpful in monitoring all of this. You can see if
> the following article helps at all.
> 
> http://java.sun.com/javase/technologies/hotspot/gc/index.jsp
> 
> -----Original Message-----
> From: Peter Cheng [mailto:petercheng456@gmail.com] 
> Sent: Sunday, October 05, 2008 7:55 AM
> To: java-user@lucene.apache.org
> Subject: RE: Memory eaten up by String, Term and TermInfo?
> 
> I have confirmed that the OutOfMemoryError is not Lucene's 
> problem. It's
> just because JVM failed to perform GC when necessary, and I don't know
> why.
> To fix this, I started another thread to call GC every six hours, and
> problems got solved.
> 
> Thank you all.
> 
> Gong
> 
> > -----Original Message-----
> > From: Michael McCandless [mailto:lucene@mikemccandless.com] 
> > Sent: Sunday, September 14, 2008 10:28 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: Memory eaten up by String, Term and TermInfo?
> > 
> > 
> > Small correction: it was checked in this morning (at least, on the  
> > East Coast of the US).
> > 
> > So you need to either build your own JAR using Lucene's trunk, or,  
> > wait for tonite's build to run and then download the build 
> artifacts  
> > from here:
> > 
> >      http://hudson.zones.apache.org/hudson/job/Lucene-trunk
> > 
> > If possible, please report back if this fixed your OutOfMemoryError.
> > 
> > 2.4 will include this fix.
> > 
> > Mike
> > 
> > Chris Lu wrote:
> > 
> > > Can you try to update to the latest Lucene svn version, like  
> > > yesterday?
> > > LUCENE-1383 was checked in yesterday. This patch is 
> > addressing a leak
> > > problem particular to J2EE applications.
> > >
> > > -- 
> > > Chris Lu
> > > -------------------------
> > > Instant Scalable Full-Text Search On Any Database/Application
> > > site: http://www.dbsight.net
> > > demo: http://search.dbsight.com
> > > Lucene Database Search in 3 minutes:
> > > 
> > http://wiki.dbsight.com/index.php?title=Create_Lucene_Database
> > _Search_in_3_minutes
> > > DBSight customer, a shopping comparison site, (anonymous per  
> > > request) got
> > > 2.6 Million Euro funding!
> > >
> > > On Sun, Sep 14, 2008 at 6:58 AM, Peter Cheng  
> > > <petercheng456@gmail.com>wrote:
> > >
> > >> Hi the community,
> > >>
> > >> In a Tomcat application (a keyword-based search engine), I use  
> > >> Lucene 2.3.2
> > >> to index 60 million documents, but each document is 
> small-sized.  
> > >> The total
> > >> index size is about 60GB.
> > >> After a successful running for a week, Tomcat was down due to  
> > >> OutOfMemory.
> > >> Then I restarted Tomcat, and after three days, I used jmap 
> > and jhat  
> > >> to find
> > >> out what had eaten up the memory. I found millions of 
> > instances of  
> > >> String,
> > >> Term, and TermInfo. Why?
> > >>
> > >> In my application, I use a single IndexSearcher object, 
> which is  
> > >> shared by
> > >> all the requests. It is opened initially, and will never 
> be closed.
> > >>
> > >> What could have eaten up the memory? What is referring to 
> > millions of
> > >> instances of Term and TermInfo?
> > >>
> > >> I can provide any snippets of codes if necessary.
> > >> Thank you so much!
> > >>
> > >> Gong Cheng
> > >>
> > >>
> > >> 
> > 
> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > >> For additional commands, e-mail: java-user-help@lucene.apache.org
> > >>
> > >>
> > 
> > 
> > 
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> > 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message