Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Subject: SegmentReader retaining memory
Date: Mon, 22 Jun 2009 18:32:50 -0400
Message-ID: <7A5CB4A7BBCE0C40B81C5145C326C31301A62971@NUMEVP06.na.imtn.com>
Thread-Topic: SegmentReader retaining memory
Thread-Index: AcnziWAOpexpyTzrSU+8qRzQeH2CGg==
From: "Groose, Brian" <Brian.Groose@ironmountain.com>
To: <java-user@lucene.apache.org>
Content-Type: text/plain;
	charset="us-ascii"

In the application I'm working on, I'm opening a new index every 15-20=0D=
=0Aminutes=2E  This is done by opening the new index and then closing the o=
ld=0D=0Aindex=2E=0D=0A=0D=0AOpening one of these indexes, about 58GB on dis=
k, appears to use about=0D=0A700MB of memory based on some simple testing=
=2E I am passing -Xmx3072m to=0D=0Athe JVM=2E=0D=0A=0D=0AI've run into some=
 OutOfMemoryErrors, and looking at the heap dump that=0D=0Awas generated, I=
 see three ReadOnlyMultiSegmentReaders, each one=0D=0Areferencing roughly 7=
00MB of heap:=0D=0A- One is referenced as the currently in-use index=2E=0D=
=0A- One is being opened and a simple query is running to force the terms=
=0D=0Ato be cached in RAM=2E=0D=0A- One is only referenced by the Finalizer=
 thread=2E=0D=0A=0D=0AI definitely have an issue where GC isn't keeping up,=
 but I'm a bit=0D=0Aconcerned that a class (DirectoryIndexReader) that has =
a finalize()=0D=0Amethod is referencing so much memory and delaying GC of t=
hat memory=0D=0Auntil the finalizer can get around to it=2E  Has anyone els=
e run into an=0D=0Aissue like this?=0D=0A=0D=0AIt seems like SegmentReader=
=2EdoClose() could null out its TermInfosReader=0D=0Ato make the large amou=
nt of cached terms available for GC much sooner=0D=0Aand not reliant on the=
 finalizer=2E=0D=0A=0D=0AI'm using Lucene 2=2E4=2E1 with Sun's JDK 6 update=
 12, 64-bit, on Debian=0D=0ALenny=2E=0D=0A=0D=0AThanks,=0D=0ABrian=0D=0AThe=
 information contained in this email message and its attachments=0Ais inten=
ded=0D=0Aonly for the private and confidential use of the recipient(s) name=
d=0Aabove, unless the sender expressly agrees otherwise=2E Transmission=0Ao=
f email over the Internet=0D=0A is not a secure communications medium=2E If=
 you are requesting or=0Ahave requested=0D=0Athe transmittal of personal da=
ta, as defined in applicable privacy=0Alaws by means=0D=0A of email or in a=
n attachment to email you must select a more=0Asecure alternate means of tr=
ansmittal that supports your=0Aobligations to protect such personal data=2E=
 If the reader of this=0Amessage is not the intended recipient and/or you h=
ave received this=0Aemail in error, you must take no action based on the in=
formation in=0Athis email and you are hereby notified that any disseminatio=
n,=0Amisuse, copying, or disclosure of this communication is strictly=0Apro=
hibited=2E If you have received=0D=0Athis communication in error, please no=
tify us immediately by email=0Aand delete the original message=2E

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org