Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 51099 invoked from network); 19 Aug 2005 17:01:19 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 19 Aug 2005 17:01:19 -0000 Received: (qmail 68529 invoked by uid 500); 19 Aug 2005 17:00:52 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 68427 invoked by uid 500); 19 Aug 2005 17:00:51 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 68386 invoked by uid 99); 19 Aug 2005 17:00:51 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 19 Aug 2005 10:00:51 -0700 X-ASF-Spam-Status: No, hits=0.1 required=10.0 tests=FORGED_RCVD_HELO,HTML_MESSAGE X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [62.173.108.19] (HELO mail21.messagelabs.com) (62.173.108.19) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 19 Aug 2005 10:01:09 -0700 X-VirusChecked: Checked X-Env-Sender: lee.turner@oyster.com X-Msg-Ref: server-8.tower-21.messagelabs.com!1124470839!26648099!1 X-StarScan-Version: 5.4.15; banners=oyster.com,-,- X-Originating-IP: [193.132.201.148] Received: (qmail 4075 invoked from network); 19 Aug 2005 17:00:39 -0000 Received: from mail.oyster.co.uk (HELO opexch2.ad.oyster.com) (193.132.201.148) by server-8.tower-21.messagelabs.com with SMTP; 19 Aug 2005 17:00:39 -0000 Received: from opexch1.ad.oyster.com ([10.10.10.83]) by opexch2.ad.oyster.com with Microsoft SMTPSVC(6.0.3790.211); Fri, 19 Aug 2005 18:00:40 +0100 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C5A4DF.87A0A53D" Subject: Use of ThreadLocal in TermInfosReader Date: Fri, 19 Aug 2005 17:59:31 +0100 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Use of ThreadLocal in TermInfosReader Thread-Index: AcWk31580oBk19mvTqCA8MqWllsTNQ== From: "Lee Turner" To: X-OriginalArrivalTime: 19 Aug 2005 17:00:40.0778 (UTC) FILETIME=[877E22A0:01C5A4DF] X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N ------_=_NextPart_001_01C5A4DF.87A0A53D Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi =20 We=20am=20currently=20investigating=20an=20issue=20I=20have=20with=20our=20= application around=20its=20memory=20consumption.=20=20We=20are=20running=20our=20web=20= application=20under load=20with=2020=20users=20and=201GB=20heap=20size=20and=20at=20around=207= =20hours=20it=20runs=20out of=20memory.=20=20We=20are=20using=20Lucene=201.4.3=20and=20JDK=201.4.2_06= . =20 We=20are=20working=20with=20a=20file=20based=20index=20of=20around=2050000= =20documents=20with frequent=20updates.=20=20On=20profiling=20the=20application=20we=20saw=20a= =20large=20number=20of ThreadLocal=20entries=20being=20created=20containing=20SegmentTermEnum's.=20= =20We tracked=20the=20use=20of=20ThreadLocals=20to=20the=20TermInfosReader=20cla= ss=20where=20we noticed=20the=20enumerator=20ThreadLocal=20wasn't=20being=20nulled.=20=20W= e=20added=20a finalize=20method=20to=20make=20sure=20this=20happened,=20which=20also=20s= eems=20to=20have been=20added=20to=20the=20latest=20source=20code=20in=20the=20lucene=20rep= ository.=20=20This=20was the=20method=20we=20added: =20 =20=20protected=20void=20finalize()=20throws=20Throwable=20{=20=20=20=20tr= y=20{ =20=20=20=20=20=20=20=20enumerators.set(null); =20=20=20=20} =20=20=20=20finally=20{ =20=20=20=20=20=20=20=20super.finalize(); =20=20=20=20} =20=20} =20 However,=20this=20didn't=20seem=20to=20clear=20up=20the=20Threadlocal's=20= that=20were hanging=20around.=20=20Upon=20further=20investigation=20of=20the=20lucene=20= source=20we realised=20that=20the=20SegmentTermEnum=20class=20has=20a=20close=20method= =20that=20doesn't seem=20to=20be=20being=20called=20before=20the=20enumerator.set(null)=20me= thod=20is called.=20=20We=20therefore=20made=20sure=20that=20this=20was=20called=20i= n=20the=20finalize=20by adding=20the=20following: =20 =20=20protected=20void=20finalize()=20throws=20Throwable=20{ =20=20=20=20//=20System.out.println("Calling=20finalize=20in=20TermInfosRe= ader"); =20=20=20=20try=20{ =20=20=20=20=20=20=20=20try=20{ =20=20=20=20=20=20=20=20=20=20=20=20getEnum().close(); =20=20=20=20=20=20=20=20} =20=20=20=20=20=20=20=20catch(IOException=20ioe) =20=20=20=20=20=20=20=20{} =20=20=20=20=20=20=20=20enumerators.set(null); =20=20=20=20} =20=20=20=20finally=20{ =20=20=20=20=20=20=20=20super.finalize(); =20=20=20=20} =20=20} =20 After=20adding=20the=20call=20to=20close()=20on=20the=20SegmentTermEnum=20= we=20saw=20the number=20of=20ThreadLocal=20entries=20drop=20dramatically.=20=20However,=20= when=20we=20now try=20to=20added=20things=20to=20the=20index=20we=20are=20getting ArrayIndexOutOfBoundsException's: =20 java.lang.ArrayIndexOutOfBoundsException:=20159 =20=20=20=20=20=20=20=20=20=20=20=20at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:142) =20=20=20=20=20=20=20=20=20=20=20=20at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:253) =20=20=20=20=20=20=20=20=20=20=20=20at org.apache.lucene.index.MultiReader.docFreq(MultiReader.java:192) =20=20=20=20=20=20=20=20=20=20=20=20at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:69) =20=20=20=20=20=20=20=20=20=20=20=20at org.apache.lucene.search.Similarity.idf(Similarity.java:255) =20=20=20=20=20=20=20=20=20=20=20=20at org.apache.lucene.search.TermQuery$TermWeight.sumOfSquaredWeights(TermQu ery.java:47) =20=20=20=20=20=20=20=20=20=20=20=20at org.apache.lucene.search.BooleanQuery$BooleanWeight.sumOfSquaredWeights( BooleanQuery.java:110) =20=20=20=20=20=20=20=20=20=20=20=20at org.apache.lucene.search.FilteredQuery$1.sumOfSquaredWeights(FilteredQue ry.java:65) =20=20=20=20=20=20=20=20=20=20=20=20at org.apache.lucene.search.BooleanQuery$BooleanWeight.sumOfSquaredWeights( BooleanQuery.java:110) =20=20=20=20=20=20=20=20=20=20=20=20at=20org.apache.lucene.search.Query.we= ight(Query.java:86) =20=20=20=20=20=20=20=20=20=20=20=20at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:85) =20=20=20=20=20=20=20=20=20=20=20=20at=20org.apache.lucene.search.Hits.get= MoreDocs(Hits.java:64) =20=20=20=20=20=20=20=20=20=20=20=20at=20org.apache.lucene.search.Hits.(Hits.java:43) =20=20=20=20=20=20=20=20=20=20=20=20at org.apache.lucene.search.Searcher.search(Searcher.java:33) =20=20=20=20=20=20=20=20=20=20=20=20at org.apache.lucene.search.Searcher.search(Searcher.java:27) =20=20=20=20=20=20=20=20=20=20=20=20at com.oyster.mom.search.SearchImpl.doSearch(SearchImpl.java:93) =20 We=20are=20really=20sorry=20for=20the=20long=20email=20and=20we=20have=20t= o=20admit=20that=20we don't=20know=20the=20lucene=20source=20very=20well=20but=20we=20were=20won= dering=20whether anyone=20could=20explain=20why=20when=20the=20SegmentTermEnum=20is=20close= d=20in=20the finalize=20it=20causes=20problems=20and=20whether=20anyone=20else=20has=20= had=20issues=20with ThreadLocals=20hanging=20around=20? =20 Any=20help=20would=20be=20greatly=20appreciated. =20 Many=20thanks Lee =20 Lee=20Turner=20|=20Java=20Developer=20|=20Oyster=20Partners=20 D.=20+44=20(0)20=2074461418 T.=20+44=20(0)20=207446=207500=20 www.oyster.com =20 __________________________________________________________________________= _______________________________________________ Internet=20communications=20are=20not=20secure=20and=20therefore=20Oyster=20= Partners=20Ltd=20does=20not=20accept=20legal=20responsibility=20for=20the=20= contents=20of=20this=20message.=20Any=20views=20or=20opinions=20presented=20= are=20solely=20those=20of=20the=20author=20and=20do=20not=20necessarily=20= represent=20those=20of=20Oyster=20Partners=20Ltd. ------_=_NextPart_001_01C5A4DF.87A0A53D--