Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: Memory eaten up by String, Term and TermInfo?
Date: Mon, 6 Oct 2008 08:48:20 -0400
Message-ID: 
 <CC2671C08F06704ABFD37ED1EA09AA56744DE5@birexchange.BIRPLAZA.local>
Thread-Topic: Memory eaten up by String, Term and TermInfo?
Thread-Index: AckWdl0PQFXQ9rc3RdKl/tcAsL+FGgQamDMwADPnjSA=
From: "Beard, Brian" <Brian.Beard@mybir.com>
To: <java-user@lucene.apache.org>

I played around with GC quite a bit in our app and found the following
java settings to help a lot (Used with jboss, but should be good for any
jvm).

set JAVA_OPTS=3D%JAVA_OPTS% -XX:MaxPermSize=3D512M =
-XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled

While these settings did help when the GC starts to get taxed for
keeping throughput better, the biggest thing was getting the heap size
big enough. If the heap's big enough, then everything seems to work well
regardless of the GC algorithm selected.

If the heap size and algorithm are correct, then you shouldn't have to
force GC. Usually the forced GC calls will trigger the total GC which
has a long delay and slows down responsiveness to the app.

I found jstat fairly helpful in monitoring all of this. You can see if
the following article helps at all.

http://java.sun.com/javase/technologies/hotspot/gc/index.jsp

-----Original Message-----
From: Peter Cheng [mailto:petercheng456@gmail.com]=20
Sent: Sunday, October 05, 2008 7:55 AM
To: java-user@lucene.apache.org
Subject: RE: Memory eaten up by String, Term and TermInfo?

I have confirmed that the OutOfMemoryError is not Lucene's problem. It's
just because JVM failed to perform GC when necessary, and I don't know
why.
To fix this, I started another thread to call GC every six hours, and
problems got solved.

Thank you all.

Gong

> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]=20
> Sent: Sunday, September 14, 2008 10:28 PM
> To: java-user@lucene.apache.org
> Subject: Re: Memory eaten up by String, Term and TermInfo?
>=20
>=20
> Small correction: it was checked in this morning (at least, on the =20
> East Coast of the US).
>=20
> So you need to either build your own JAR using Lucene's trunk, or, =20
> wait for tonite's build to run and then download the build artifacts =20
> from here:
>=20
>      http://hudson.zones.apache.org/hudson/job/Lucene-trunk
>=20
> If possible, please report back if this fixed your OutOfMemoryError.
>=20
> 2.4 will include this fix.
>=20
> Mike
>=20
> Chris Lu wrote:
>=20
> > Can you try to update to the latest Lucene svn version, like =20
> > yesterday?
> > LUCENE-1383 was checked in yesterday. This patch is=20
> addressing a leak
> > problem particular to J2EE applications.
> >
> > --=20
> > Chris Lu
> > -------------------------
> > Instant Scalable Full-Text Search On Any Database/Application
> > site: http://www.dbsight.net
> > demo: http://search.dbsight.com
> > Lucene Database Search in 3 minutes:
> >=20
> http://wiki.dbsight.com/index.php?title=3DCreate_Lucene_Database
> _Search_in_3_minutes
> > DBSight customer, a shopping comparison site, (anonymous per =20
> > request) got
> > 2.6 Million Euro funding!
> >
> > On Sun, Sep 14, 2008 at 6:58 AM, Peter Cheng =20
> > <petercheng456@gmail.com>wrote:
> >
> >> Hi the community,
> >>
> >> In a Tomcat application (a keyword-based search engine), I use =20
> >> Lucene 2.3.2
> >> to index 60 million documents, but each document is small-sized. =20
> >> The total
> >> index size is about 60GB.
> >> After a successful running for a week, Tomcat was down due to =20
> >> OutOfMemory.
> >> Then I restarted Tomcat, and after three days, I used jmap=20
> and jhat =20
> >> to find
> >> out what had eaten up the memory. I found millions of=20
> instances of =20
> >> String,
> >> Term, and TermInfo. Why?
> >>
> >> In my application, I use a single IndexSearcher object, which is =20
> >> shared by
> >> all the requests. It is opened initially, and will never be closed.
> >>
> >> What could have eaten up the memory? What is referring to=20
> millions of
> >> instances of Term and TermInfo?
> >>
> >> I can provide any snippets of codes if necessary.
> >> Thank you so much!
> >>
> >> Gong Cheng
> >>
> >>
> >>=20
> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
>=20
>=20
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>=20


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org