Return-Path: X-Original-To: apmail-lucene-java-user-archive@www.apache.org Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C8AB5CEF4 for ; Wed, 16 May 2012 16:01:22 +0000 (UTC) Received: (qmail 4562 invoked by uid 500); 16 May 2012 16:01:20 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 4466 invoked by uid 500); 16 May 2012 16:01:20 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 4458 invoked by uid 99); 16 May 2012 16:01:20 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 May 2012 16:01:20 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of chris.bamford@talktalk.net designates 62.24.128.245 as permitted sender) Received: from [62.24.128.245] (HELO out1.ip09ir2.opaltelecom.net) (62.24.128.245) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 May 2012 16:01:13 +0000 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArMEAALOs0/DXQQl/2dsb2JhbABEgkWnfAGIPoISghUBAQUtIR4KDwwDEQQBAQEnOBUJCA4HBAEcBIdtBAeZdaEzixMZhTwDlwuEYIpPgwWBQw X-IronPort-AV: E=Sophos;i="4.75,604,1330905600"; d="scan'208";a="514709994" Received: from webmail-vfrr17.sis.aol.com (HELO webmail-vfrr17) ([195.93.4.37]) by out1.ip09ir2.opaltelecom.net with ESMTP; 16 May 2012 17:00:52 +0100 To: java-user@lucene.apache.org Subject: Re: Memory question Date: Wed, 16 May 2012 12:00:52 -0400 In-Reply-To: X-MB-Message-Source: WebUI MIME-Version: 1.0 From: Chris Bamford X-MB-Message-Type: User Content-Type: multipart/alternative; boundary="--------MB_8CF0194342881A7_12D4_3F123_webmail-vfrr17.sis.aol.com" X-Mailer: Webmail 34561-STANDARD Received: from 146.101.57.209 by webmail-vfrr17.sis.aol.com (195.93.4.37) with HTTP (WebMailUI); Wed, 16 May 2012 12:00:32 -0400 Message-Id: <8CF019427FA3867-12D4-1141D@webmail-vfrr17.sis.aol.com> ----------MB_8CF0194342881A7_12D4_3F123_webmail-vfrr17.sis.aol.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="us-ascii" =0A=20Thanks=20everyone.=20Looks=20like=20I=20have=20lots=20of=20reading=20to=20= do=20:-)=0A=0A=20=0A=0A=20=0A=0A-----Original=20Message-----=0AFrom:=20Nader,=20= John=20P=20=0ATo:=20java-user@lucene.apache.org=20=0ASent:=20Wed,=2016=20May=202012=2016:27=0ASubject:=20Re= :=20Memory=20question=0A=0A=0AAnother=20good=20link=20is=0Ahttp://www.oracle.com= /technetwork/java/javase/gc-tuning-6-140523.html,=0Awhich=20also=20includes=20de= tails=20on=20iCMS,=20which=20is=20the=20Incremental=20Mode=20for=20CMS.=0A=0AOn=20= 5/15/12=206:32=20PM,=20"Lutz=20Fechner"=20=20wrote:=0A=0A>C= MS=20is=20the=20concurrent=20mark=20sweep=20garbage=20collector.=20Instead=20of=20= waiting=0A>for=20the=20memory=20to=20fill=20up=20and=20being=20collected=20for=20= memory=20to=20be=20freed=20up=0A>again,=20it=20runs=20concurrently=20while=20the= =20app=20threads=20are=20running.=20Usually=0A>the=20JVM=20would=20call=20a=20fu= ll=20stop=20the=20world=20collection.=20All=20threads=20would=0A>be=20on=20hold=20= until=20GC=20is=20finished.=20CMS=20would=20prevent=20lengthy=20stops=20from=0A>= happening=20in=20trade=20for=20constant=20micro=20stops.=0A>=0A>This=20usually=20= is=20a=20good=20option=20for=20apps=20that=20are=20sensitive=20(from=20a=20user=0A= >experience)=20from=20hanging=20during=20GC=20time.=0A>=0A>See=20http://docs.ora= cle.com/javase/6/docs/technotes/guides/vm/cms-6.html=0A>=0A>Best=20Regards=0A>=0A= >Lutz=0A>=0A>-----Original=20Message-----=0A>From:=20Chris=20Bamford=20[mailto:c= hris.bamford@talktalk.net]=0A>Sent:=20Dienstag,=2015.=20Mai=202012=2016:38=0A>To= :=20java-user@lucene.apache.org=0A>Subject:=20Re:=20Memory=20question=0A>=0A>=0A= >=20Hi=20John,=0A>=0A>Very=20interesting,=20thanks=20for=20the=20detailed=20expl= anation.=20=20=20It=20certainly=0A>sounds=20like=20the=20same=20symptoms!=0A>Can= =20I=20please=20clarify=20a=20couple=20of=20things=20?=0A>=0A>-=20I=20have=20goo= gled=20CMS=20/=20iCMS=20as=20I=20wasn't=20familiar=20with=20those=20acronyms=0A>= (apart=20from=20'code=20management=20system'=20and=20that=20didn't=20sound=20rig= ht!)=0A>=20=20and=20am=20I=20right=20in=20thinking=20that=20it=20is=20some=20sor= t=20of=20monitoring=20code=0A>pulled=20into=20your=20server=20via=20a=20jar?=20(= I'm=20confused=20why=20it=20would=20have=20its'=0A>own=0A>=20=20GC=20cycle...)=0A= >-=20So=20are=20you=20suggesting=20I=20play=20with=20my=20own=20JVM's=20(Sun/Ora= cle)=20parameters=0A>to=20achieve=20a=20similar=20effect=20?=0A>=0A>Thanks=20aga= in,=0A>=0A>-=20Chris=0A>=0A>=0A>=20=0A>=0A>=20=0A>=0A>-----Original=20Message---= --=0A>From:=20Nader,=20John=20P=20=0A>To:=20java-user@lu= cene.apache.org=20=0A>Sent:=20Tue,=2015=20May=20201= 2=2021:12=0A>Subject:=20Re:=20Memory=20question=0A>=0A>=0A>We've=20encountered=20= this=20issue=20and=20came=20up=20with=20a=20fairly=20good=20approach=20to=0A>add= ress=20it.=0A>=0A>We=20are=20on=20Lucene=203.0.2=20with=20Java=201.6.0_29.=20=20= Our=20indices=20are=20about=2035GB=0A>in=0A>size.=20=20Our=20JVM=20runs=20at=202= 0GB=20of=20heap,=20with=20about=2012GB=20of=20steady=20usage.=0A>Our=0A>server=20= has=2032GB=20total.=0A>=0A>What=20would=20happen=20in=20our=20case=20is=20that=20= the=20linux=20would=20page=20in=20more=20and=0A>more=20of=20the=20memory=20mappe= d=20index=20files=20into=20memory,=20forcing=20idle=20portions=0A>of=20the=20JVM= =20heap=20to=20be=20swapped=20out.=20=20This=20was=20not=20an=20issue=20until=20= our=20CMS=0A>GC=0A>kicked=20in.=20=20This=20would=20force=20swapping=20in=20of=20= all=20JVM=20memory=20to=20collect=0A>unused=20references.=20=20I/O=20wait=20woul= d=20shoot=20up=20and=20performance=20would=0A>suffer.=0A>Yes,=20even=20CMS=20can= =20kill=20performance=20is=20you=20are=20swapping.=20=20The=20tell-tale=0A>sign=20= was=20a=20spike=20in=20inbound=20swap=20at=20the=20start=20of=20CMS.=0A>=0A>In=20= our=20case,=20we=20addressed=20the=20situation=20using=20iCMS,=20which=20is=20In= cremental=0A>CMS.=20=20This=20takes=20the=20mark=20phase=20(and=20sweep=20too?)=20= and=20does=20it=0A>continuously=0A>with=20a=20configurable=20duty=20cycle.=20=20= The=20result=20was=20that=20swapping=20was=0A>smoothed=0A>out=20to=20be=20a=20sm= all=20stead=20drag=20on=20the=20system=20instead=20of=20a=20hard=20spike.=0A>The= re=0A>was=20an=20small=20loss=20in=20performance,=20but=20a=20big=20gain=20in=20= stability.=0A>=0A>This=20tuning=20may=20be=20an=20option=20for=20you.=20=20BTW,=20= pmap=20will=20give=20you=20statistic=0A>on=20total=20file=20size=20and=20how=20m= uch=20is=20resident.=20=20The=20java=20heap=20shows=20up=20in=0A>pmap=20as=20wel= l=20on=20linux,=20so=20you=20can=20determine=20how=20much=20of=20that=20is=20in=0A= >memory=0A>as=20well.=0A>=0A>John=0A>=0A>=0A>=0A>On=205/15/12=203:38=20PM,=20"Ch= ris=20Bamford"=20=20wrote:=0A>=0A>>Thanks=20Uwe.=20=0A= >>=0A>>What=20I'd=20like=20to=20understand=20is=20the=20implications=20of=20this= =20on=20a=20server=0A>which=0A>>opens=20a=20large=20number=20of=20indexes=20over= =20a=20long=20period.=20Will=20this=20non-heap=0A>>memory=20continue=20to=20grow= ?=20Will=20gc=20be=20effective=20at=20spotting=20it=20and=0A>>releasing=20it=20v= ia=20references=20in=20the=20heap?=0A>>=0A>>=20I=20had=20an=20instance=20yesterd= ay=20where=20a=20server=20swapped=20itself=20to=20a=0A>>standstill=20and=20had=20= to=20be=20restarted.=20The=20load=20average=20was=20through=20the=0A>roof=0A>>an= d=20I=20am=20trying=20to=20understand=20why.=20One=20of=20my=20recent=20changes=20= is=20updating=0A>>from=202.3=20to=203.6,=20so=20naturally=20I=20am=20keen=20to=20= know=20the=20impact=20of=20the=20mmap=0A>>stuff=20which=20is=20now=20standard=20= under=20the=20covers.=0A>>=0A>>My=20server=20caches=20indexsearchers=20and=20the= n=20closes=20them=20based=20on=20how=20full=0A>>the=20heap=20is=20getting.=20My=20= worry=20is=20that=20if=20the=20bulk=20of=20the=20memory=20is=0A>being=0A>>alloca= ted=20outside=20the=20Jvm,=20how=20can=20I=20make=20sensible=20decisions?=0A>>=0A= >>Thanks=20for=20any=20pointers=20/=20info.=0A>>=0A>>Chris=0A>>=0A>>=0A>>=0A>>--= ---Original=20Message-----=0A>>From:=20uwe@thetaphi.de=0A>>To:=20java-user@lucen= e.apache.org=0A>>Sent:=20Tue,=2015=20May=202012=2018:10=0A>>Subject:=20RE:=20Mem= ory=20question=0A>>=0A>>=0A>>=0A>>It=20mmaps=20the=20files=20into=20virtual=20me= mory=20if=20it=20runs=20on=20a=2064=20bit=20JVM.=0A>Because=0A>>of=20that=20you=20= see=20the=20mmapped=20CFS=20files.=20This=20is=20outside=20Java=20Heap=20and=20i= s=0A>>all=0A>>*virtual*=20no=20RAM=20is=20explicitely=20occupied=20except=20the=20= O/S=20cache.=0A>>=0A>>-----=0A>>Uwe=20Schindler=0A>>H.-H.-Meier-Allee=2063,=20D-= 28213=20Bremen=0A>>http://www.thetaphi.de=0A>>eMail:=20uwe@thetaphi.de=0A>>=0A>>= >=20-----Original=20Message-----=0A>>>=20From:=20Chris=20Bamford=20[mailto:chris= .bamford@talktalk.net]=0A>>>=20Sent:=20Tuesday,=20May=2015,=202012=204:47=20PM=0A= >>>=20To:=20java-user@lucene.apache.org=0A>>>=20Subject:=20Memory=20question=0A>= >>=20=0A>>>=20Hi=0A>>>=20=0A>>>=20Can=20anyone=20tell=20me=20what=20happens=20to= =20the=20memory=20when=20Lucene=20opens=20an=0A>>>index?=0A>>>=20Is=20it=20loade= d=20into=20the=20JVM's=20heap=20or=20is=20it=20mapped=20into=20virtual=20memory=0A= >>outside=20of=0A>>>=20it?=0A>>>=20I=20am=20running=20on=20Linux=20and=20if=20I=20= use=20pmap=20on=20the=20PID=20of=20my=20JVM,=20I=20can=0A>see=0A>>lots=20of=0A>>= >=20entries=20for=20index=20cfs=20files.=0A>>>=20=0A>>>=20Does=20this=20mean=20t= hat=20indexes=20are=20mapped=20into=20non-heap=20memory?=20=20If=20so,=0A>how=0A= >>>=20can=20I=20monitor=20the=20space=20my=20process=20is=20using=20if=20I=20cac= he=20open=0A>>IndexSearchers?=0A>>>=20=0A>>>=20The=20details=20are:=0A>>>=20=0A>= >>=20Sun=2064-bit=20JVM=20on=20Linux.=0A>>>=20Lucene=203.6=20running=20in=202.3=20= compatibility=20mode=20(as=20we=20are=20in=20the=20in=20the=0A>>process=20of=0A>= >>=20a=20migration=20to=203.6)=0A>>>=20=0A>>>=20Thanks,=0A>>>=20=0A>>>=20-=20Chr= is=0A>>=0A>>--------------------------------------------------------------------= -=0A>>To=20unsubscribe,=20e-mail:=20java-user-unsubscribe@lucene.apache.org=0A>>= For=20additional=20commands,=20e-mail:=20java-user-help@lucene.apache.org=0A>>=0A= >=0A>=0A>---------------------------------------------------------------------=0A= >To=20unsubscribe,=20e-mail:=20java-user-unsubscribe@lucene.apache.org=0A>For=20= additional=20commands,=20e-mail:=20java-user-help@lucene.apache.org=0A>=0A>=0A>=20= =0A>=0A>=0A>--------------------------------------------------------------------= -=0A>To=20unsubscribe,=20e-mail:=20java-user-unsubscribe@lucene.apache.org=0A>Fo= r=20additional=20commands,=20e-mail:=20java-user-help@lucene.apache.org=0A>=0A=0A= =0A---------------------------------------------------------------------=0ATo=20= unsubscribe,=20e-mail:=20java-user-unsubscribe@lucene.apache.org=0AFor=20additio= nal=20commands,=20e-mail:=20java-user-help@lucene.apache.org=0A=0A=0A=20=0A ----------MB_8CF0194342881A7_12D4_3F123_webmail-vfrr17.sis.aol.com--