Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CC99B95AE for ; Tue, 1 May 2012 15:58:20 +0000 (UTC) Received: (qmail 51144 invoked by uid 500); 1 May 2012 15:58:19 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 51109 invoked by uid 500); 1 May 2012 15:58:18 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 51096 invoked by uid 99); 1 May 2012 15:58:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 May 2012 15:58:18 +0000 X-ASF-Spam-Status: No, hits=0.7 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [98.139.91.86] (HELO nm16.bullet.mail.sp2.yahoo.com) (98.139.91.86) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 01 May 2012 15:58:13 +0000 Received: from [98.139.91.68] by nm16.bullet.mail.sp2.yahoo.com with NNFMP; 01 May 2012 15:57:52 -0000 Received: from [98.139.91.15] by tm8.bullet.mail.sp2.yahoo.com with NNFMP; 01 May 2012 15:57:52 -0000 Received: from [127.0.0.1] by omp1015.mail.sp2.yahoo.com with NNFMP; 01 May 2012 15:57:52 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 810634.16869.bm@omp1015.mail.sp2.yahoo.com Received: (qmail 40938 invoked by uid 60001); 1 May 2012 15:57:52 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1335887872; bh=Xu1ztLfjDOmomnNXqtNcirgJ/KIEJqjqtLH7m4Pcwfk=; h=X-YMail-OSG:Received:X-RocketYMMF:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=E+sRY9Nn9vu2zlgKLBpqzq30pLAfYz6Y2Sq0qlL6bh0oMGEomk5n6uVWbtB4YsAAmmYTklAQ9NyTGLOAQGK8LA85tgeW538i8A6+4d+TUF636NID72aZwKP7DsQXmHO7PgDNSmDeLetYU24GtNOE/sPEsGeE+qaErRqAkSR+s2c= X-YMail-OSG: jrfGl0oVM1llw1lAZxTqOmNuxid7b7fo6YRIdZVJUPeCKny fjmNPKmBFDXsseVyMzPK2oEDXzgh96EtMNB2OUSH6kTaApemBiUTNNf5L2Ws l6AGico0pFuxougQOixgMxHDtlqfjMIPWNyALvvghyIu0lZdzq_MysKNXf2h R5GuVSvU.O12Ul2mlDxWnxh7ZBA94MPr7LmzsvLNPNNSfL5lfUyLpD0pBiW1 aKq4sqXEPqyr2M6znMo4HhjWzpl4tGsDzO.orVYJm5ht1.hED2YErM2Lb677 aZ34KKRM_fcVPX5e889zPP1m_mdY1E7UBBqpwFA4DLR4mGUWMYjeWxb63JVx 2NQfo5YUDmSL1kRALBaPUVB3s3aWXXmPy6mfcIeJL.4TuCkcYWihD0P7zCe. B4JbxvSCxA9CwOKVLJqQ9n0nA9kSopmgTG844sqgHb8rUKP6bLQ3koSFEXp4 jdqyuN3qeHu6nh02abKeR4wyiMxjFw.cgfebwyPv0K0ZIRZNQj6Utfg31_eh mbay.5BHB.vW2rxQv5IS1xOJGoD2PaJbWdvMSFccWPi7q0kSfpLBVsPV72p9 l2RxYirXftcDRXMlNOyWKQqIvRP6g1lwQXOUFkaZq85woKkHGRAjzE4TOlJH ofBtswBYc9a1Lfum5H1dPMeirCObVKVWg3A4Qw1dLBC6qzKMgKRDOzSaEVup 6TBLU.8c2iETHVWcigpz9dDlJlgG.N9IkJgs- Received: from [173.200.178.70] by web164502.mail.gq1.yahoo.com via HTTP; Tue, 01 May 2012 08:57:51 PDT X-RocketYMMF: apurtell X-Mailer: YahooMailWebService/0.8.117.340979 References: <1335877356.88800.BPMail_high_noncarrier@web192501.mail.sg3.yahoo.com> Message-ID: <1335887871.31256.YahooMailNeo@web164502.mail.gq1.yahoo.com> Date: Tue, 1 May 2012 08:57:51 -0700 (PDT) From: Andrew Purtell Reply-To: Andrew Purtell Subject: Re: RegionServer silently stops (only "issue": CMS-concurrent-mark ~80sec) To: "user@hbase.apache.org" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Does it make sense to include the below in ./bin/hbase and ./bin/hadoop:=0A= =0A=A0=A0=A0 echo -17 > /proc/$PID/oom_adj=0A=0A-17 is OOM_DISABLE: http://= linux-mm.org/OOM_Killer=0A=0ABest regards,=0A=0A=A0 =A0 - Andy=0A=0AProblem= s worthy of attack prove their worth by hitting back. - Piet Hein (via Tom = White)=0A=0A=0A=0A----- Original Message -----=0A> From: Alex Baranau =0A> To: user@hbase.apache.org=0A> Cc: =0A> Sent: Tues= day, May 1, 2012 8:27 AM=0A> Subject: Re: RegionServer silently stops (only= "issue": CMS-concurrent-mark ~80sec)=0A> =0A> Yep, that is what I thought = (OOM). Our monitoring tool (sematext.com/spm)=0A> says that everything was = OK with theregard to JVM memory consumption at the=0A> time process stopped= . But found that oom killer is one who killed it (on=0A> this cluster we ha= ve HBASE_HEAPSIZE=3D4000):=0A> =0A> /var/log/kern.log.1:Apr 30 18:59:04 ip-= 10-88-231-73 kernel:=0A> [32076201.925847] Out of memory: Kill process 1522= 7 (java) score 573 or=0A> sacrifice child=0A> /var/log/kern.log.1-Apr 30 18= :59:04 ip-10-88-231-73 kernel:=0A> [32076201.925881] Killed process 15227 (= java) total-vm:4716576kB,=0A> anon-rss:4364780kB, file-rss:12020kB=0A> =0A>= All clear, thanx!=0A> =0A> Alex Baranau=0A> ------=0A> Sematext :: http://= blog.sematext.com/=0A> =0A> On Tue, May 1, 2012 at 9:02 AM, Dhaval Shah =0A= > wrote:=0A> =0A>> =0A>> =0A>>=A0=A0Not sure i= f its related (or even helpful) but we were using cdh3b4 (which=0A>>=A0=A0i= s 0.90.1) and we saw similar issues with region servers going down.. we=0A>= >=A0=A0didn't look at GC logs but we had very high zookeeper leases so its= =0A>>=A0=A0unlikely that the GC could have caused the issue.. this problem = went away=0A>>=A0=A0when we upgraded to cdh3u3 which is rock steady in term= s of region=0A>>=A0=A0servers.. (havent had a single region server crash in= a month where on the=0A>>=A0=A0older version I used to have 1 crash every = couple of days).. the only other=0A>>=A0=A0difference between the two is th= at we use snappy on the newer one and gz on=0A>>=A0=A0the old=0A>> =0A>>=A0= =A0We also noticed that having replication enabled also contributed to the= =0A>>=A0=A0issues..=0A>> =0A>> =0A>>=A0=A0------------------------------=0A= >>=A0=A0On Tue 1 May, 2012 3:15 PM IST N Keywal wrote:=0A>> =0A>>=A0=A0>Hi = Alex,=0A>>=A0=A0>=0A>>=A0=A0>On the same idea, note that hbase is launched = with=0A>>=A0=A0>-XX:OnOutOfMemoryError=3D"kill -9 %p".=0A>>=A0=A0>=0A>>=A0= =A0>N.=0A>>=A0=A0>=0A>>=A0=A0>On Tue, May 1, 2012 at 10:41 AM, Igal Shilman= =0A> wrote:=0A>>=A0=A0>=0A>>=A0=A0>> Hi Alex, just to rule= out, oom killer,=0A>>=A0=A0>> Try this:=0A>>=A0=A0>>=0A>>=A0=A0>>=0A>> =0A= > http://stackoverflow.com/questions/624857/finding-which-process-was-kille= d-by-linux-oom-killer=0A>>=A0=A0>>=0A>>=A0=A0>>=0A>>=A0=A0>> On Mon, Apr 30= , 2012 at 10:48 PM, Alex Baranau <=0A>>=A0=A0alex.baranov.v@gmail.com=0A>>= =A0=A0>> >wrote:=0A>>=A0=A0>>=0A>>=A0=A0>> > Hello,=0A>>=A0=A0>> >=0A>>=A0= =A0>> > During recent weeks I constantly see some RSs *silently* =0A> dying= on our=0A>>=A0=A0>> HBase=0A>>=A0=A0>> > cluster. By "silently" I mean tha= t process stops, =0A> but no errors in=0A>>=A0=A0logs=0A>>=A0=A0>> > [1].= =0A>>=A0=A0>> >=0A>>=A0=A0>> > The only thing I can relate to it is long = =0A> CMS-concurrent-mark: almost=0A>>=A0=A080=0A>>=A0=A0>> > seconds. But t= his should not cause issues as it is not a=0A>>=A0=A0"stop-the-world"=0A>>= =A0=A0>> > process.=0A>>=A0=A0>> >=0A>>=A0=A0>> > Any advice?=0A>>=A0=A0>> = >=0A>>=A0=A0>> > HBase: hbase-0.90.4-cdh3u3=0A>>=A0=A0>> > Hadoop: 0.20.2-c= dh3u3=0A>>=A0=A0>> >=0A>>=A0=A0>> > Thank you,=0A>>=A0=A0>> > Alex Baranau= =0A>>=A0=A0>> >=0A>>=A0=A0>> > [1]=0A>>=A0=A0>> >=0A>>=A0=A0>> > last lines= from RS log (no errors before too, and nothing =0A> written in=0A>>=A0=A0>= > *.out=0A>>=A0=A0>> > file):=0A>>=A0=A0>> >=0A>>=A0=A0>> > 2012-04-30 18:5= 2:11,806 DEBUG=0A>>=A0=A0>> > org.apache.hadoop.hbase.regionserver.CompactS= plitThread: =0A> Compaction=0A>>=A0=A0>> > requested for agg-sa-1.3,0011|= =0A>>=A0=A0>> >=0A>>=A0=A0>> >=0A>>=A0=A0>>=0A>> =0A> te|dtc|\x00\x00\x00\x= 00\x00\x00<\x1E\x002\x00\x00\x00\x015\x9C_n\x00\x00\x00\x00\x00\x00\x00\x00= \x00,1334852280902.4285f9339b520ee617c087c0fd0dbf65.=0A>>=A0=A0>> > because= regionserver60020.cacheFlusher; priority=3D-1, =0A> compaction queue=0A>>= =A0=A0>> > size=3D0=0A>>=A0=A0>> > 2012-04-30 18:54:58,779 DEBUG=0A>>=A0=A0= >> > =0A> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: u= sing=0A>>=A0=A0new=0A>>=A0=A0>> > createWriter -- HADOOP-6840=0A>>=A0=A0>> = > 2012-04-30 18:54:58,779 DEBUG=0A>>=A0=A0>> > =0A> org.apache.hadoop.hbase= .regionserver.wal.SequenceFileLogWriter:=0A>>=A0=A0>> >=0A>>=A0=A0>> >=0A>>= =A0=A0>>=0A>> =0A> Path=3Dhdfs://xxx.ec2.internal/hbase/.logs/xxx.ec2.inter= nal,60020,1335706613397/xxx.ec2.internal%3A60020.1335812098651,=0A>>=A0=A0>= > > syncFs=3Dtrue, hflush=3Dfalse=0A>>=A0=A0>> > 2012-04-30 18:54:58,874 IN= FO=0A>>=A0=A0>> org.apache.hadoop.hbase.regionserver.wal.HLog:=0A>>=A0=A0>>= > Roll=0A>>=A0=A0>> >=0A>>=A0=A0>> >=0A>>=A0=A0>>=0A>> =0A> /hbase/.logs/x= xx.ec2.internal,60020,1335706613397/xxx.ec2.internal%3A60020.1335811856672,= =0A>>=A0=A0>> > entries=3D73789, filesize=3D63773934. New hlog=0A>>=A0=A0>>= >=0A>>=A0=A0>> >=0A>>=A0=A0>>=0A>> =0A> /hbase/.logs/xxx.ec2.internal,6002= 0,1335706613397/xxx.ec2.internal%3A60020.1335812098651=0A>>=A0=A0>> > 2012-= 04-30 18:56:31,867 INFO=0A>>=A0=A0>> > org.apache.hadoop.hbase.regionserver= .MemStoreFlusher: Flush =0A> thread=0A>>=A0=A0woke=0A>>=A0=A0>> up=0A>>=A0= =A0>> > with memory above low water.=0A>>=A0=A0>> > 2012-04-30 18:56:31,867= INFO=0A>>=A0=A0>> > org.apache.hadoop.hbase.regionserver.MemStoreFlusher: = Flush =0A> of region=0A>>=A0=A0>> > agg-sa-1.3,s_00I4|=0A>>=A0=A0>> >=0A>>= =A0=A0>> >=0A>>=A0=A0>>=0A>> =0A> tdqc\x00docs|mrtdocs|\x00\x00\x00\x00\x00= \x03\x11\xF4\x00none\x00|1334692562\x00\x0D\xE0\xB6\xB3\xA7c\xFF\xBC|268373= 73\x00\x00\x00\x016\xC1\xE0D\xBE\x00\x00\x00\x00\x00\x00\x00\x00,1335761291= 026.30b127193485342359eadf1586819805.=0A>>=A0=A0>> > due to global heap pre= ssure=0A>>=A0=A0>> > 2012-04-30 18:56:31,867 DEBUG=0A>>=A0=A0>> org.apache.= hadoop.hbase.regionserver.HRegion:=0A>>=A0=A0>> > Started memstore flush fo= r agg-sa-1.3,s_00I4|=0A>>=A0=A0>> >=0A>>=A0=A0>> >=0A>>=A0=A0>>=0A>> =0A> t= dqc\x00docs|mrtdocs|\x00\x00\x00\x00\x00\x03\x11\xF4\x00none\x00|1334692562= \x00\x0D\xE0\xB6\xB3\xA7c\xFF\xBC|26837373\x00\x00\x00\x016\xC1\xE0D\xBE\x0= 0\x00\x00\x00\x00\x00\x00\x00,1335761291026.30b127193485342359eadf158681980= 5.,=0A>>=A0=A0>> > current region memstore size 138.1m=0A>>=A0=A0>> > 2012-= 04-30 18:56:31,867 DEBUG=0A>>=A0=A0>> org.apache.hadoop.hbase.regionserver.= HRegion:=0A>>=A0=A0>> > Finished snapshotting, commencing flushing stores= =0A>>=A0=A0>> > 2012-04-30 18:56:56,303 DEBUG=0A>>=A0=A0>> > org.apache.had= oop.hbase.io.hfile.LruBlockCache: LRU Stats:=0A>>=A0=A0total=3D322.84=0A>>= =A0=A0>> MB,=0A>>=A0=A0>> > free=3D476.34 MB, max=3D799.17 MB, blocks=3D502= 4, =0A> accesses=3D12189396,=0A>>=A0=A0>> hits=3D127592,=0A>>=A0=A0>> > hit= Ratio=3D1.04%%, cachingAccesses=3D132480, cachingHits=3D126949,=0A>>=A0=A0>= > > cachingHitsRatio=3D95.82%%, evictions=3D0, evicted=3D0, =0A> evictedPer= Run=3DNaN=0A>>=A0=A0>> > 2012-04-30 18:56:59,026 INFO=0A>>=A0=A0org.apache.= hadoop.hbase.regionserver.Store:=0A>>=A0=A0>> > Renaming flushed file at=0A= >>=A0=A0>> >=0A>>=A0=A0>> >=0A>>=A0=A0>>=0A>> =0A> hdfs://zzz.ec2.internal/= hbase/agg-sa-1.3/30b127193485342359eadf1586819805/.tmp/391890051647401997= =0A>>=A0=A0>> > to=0A>>=A0=A0>> >=0A>>=A0=A0>> >=0A>>=A0=A0>>=0A>> =0A> hdf= s://zzz.ec2.internal/hbase/agg-sa-1.3/30b127193485342359eadf1586819805/a/11= 39737908876846168=0A>>=A0=A0>> > 2012-04-30 18:56:59,034 INFO=0A>>=A0=A0org= .apache.hadoop.hbase.regionserver.Store:=0A>>=A0=A0>> > Added=0A>>=A0=A0>> = >=0A>>=A0=A0>> >=0A>>=A0=A0>>=0A>> =0A> hdfs://zzz.ec2.internal/hbase/agg-s= a-1.3/30b127193485342359eadf1586819805/a/1139737908876846168,=0A>>=A0=A0>> = > entries=3D476418, sequenceid=3D880198761, memsize=3D138.1m, =0A> filesize= =3D5.7m=0A>>=A0=A0>> > 2012-04-30 18:56:59,097 INFO=0A>>=A0=A0>> org.apache= .hadoop.hbase.regionserver.HRegion:=0A>>=A0=A0>> > Finished memstore flush = of ~138.1m for region =0A> agg-sa-1.3,s_00I4|=0A>>=A0=A0>> >=0A>>=A0=A0>> >= =0A>>=A0=A0>>=0A>> =0A> tdqc\x00docs|mrtdocs|\x00\x00\x00\x00\x00\x03\x11\x= F4\x00none\x00|1334692562\x00\x0D\xE0\xB6\xB3\xA7c\xFF\xBC|26837373\x00\x00= \x00\x016\xC1\xE0D\xBE\x00\x00\x00\x00\x00\x00\x00\x00,1335761291026.30b127= 193485342359eadf1586819805.=0A>>=A0=A0>> > in 27230ms, sequenceid=3D8801987= 61, compaction requested=3Dfalse=0A>>=A0=A0>> > ~=0A>>=A0=A0>> >=0A>>=A0=A0= >> > [2]=0A>>=A0=A0>> >=0A>>=A0=A0>> > last lines from GC log:=0A>>=A0=A0>>= >=0A>>=A0=A0>> > 2012-04-30T18:58:46.683+0000: 105717.791: [GC 105717.791:= =0A> [ParNew:=0A>>=A0=A0>> > 35638K->1118K(38336K), 0.0548970 secs] =0A> 3= 145651K->3111412K(4091776K)=0A>>=A0=A0>> > icms_dc=3D6 , 0.0550360 secs] [T= imes: user=3D0.08 sys=3D0.00, =0A> real=3D0.09=0A>>=A0=A0secs]=0A>>=A0=A0>>= > 2012-04-30T18:58:46.961+0000: 105718.069: [GC 105718.069: =0A> [ParNew:= =0A>>=A0=A0>> > 35230K->2224K(38336K), 0.0802440 secs] =0A> 3145524K->31125= 33K(4091776K)=0A>>=A0=A0>> > icms_dc=3D6 , 0.0803810 secs] [Times: user=3D0= .06 sys=3D0.00, =0A> real=3D0.13=0A>>=A0=A0secs]=0A>>=A0=A0>> > 2012-04-30T= 18:58:47.114+0000: 105718.222: =0A> [CMS-concurrent-mark:=0A>>=A0=A0>> > 8.= 770/80.230 secs] [Times: user=3D61.34 sys=3D5.69, real=3D80.23 =0A> secs]= =0A>>=A0=A0>> > 2012-04-30T18:58:47.114+0000: 105718.222:=0A>>=A0=A0[CMS-co= ncurrent-preclean-start]=0A>>=A0=A0>> > 2012-04-30T18:58:47.183+0000: 10571= 8.291: [GC 105718.291: =0A> [ParNew:=0A>>=A0=A0>> > 36336K->3895K(38336K), = 0.0272610 secs] =0A> 3146645K->3114281K(4091776K)=0A>>=A0=A0>> > icms_dc=3D= 6 , 0.0274040 secs] [Times: user=3D0.05 sys=3D0.00, =0A> real=3D0.03=0A>>= =A0=A0secs]=0A>>=A0=A0>> > 2012-04-30T18:58:47.452+0000: 105718.560: [GC 10= 5718.560: =0A> [ParNew:=0A>>=A0=A0>> > 37990K->3082K(38336K), 0.0504660 sec= s] =0A> 3148376K->3114329K(4091776K)=0A>>=A0=A0>> > icms_dc=3D6 , 0.0506020= secs] [Times: user=3D0.06 sys=3D0.00, =0A> real=3D0.05=0A>>=A0=A0secs]=0A>= >=A0=A0>> > 2012-04-30T18:58:47.823+0000: 105718.930: [GC 105718.931: =0A> = [ParNew:=0A>>=A0=A0>> > 37194K->3340K(38336K), 0.0419660 secs] =0A> 3148441= K->3115727K(4091776K)=0A>>=A0=A0>> > icms_dc=3D6 , 0.0421110 secs] [Times: = user=3D0.05 sys=3D0.00, =0A> real=3D0.04=0A>>=A0=A0secs]=0A>>=A0=A0>> > 201= 2-04-30T18:58:48.099+0000: 105719.207: [GC 105719.207: =0A> [ParNew:=0A>>= =A0=A0>> > 37351K->3249K(38336K), 0.0695950 secs] =0A> 3149738K->3115636K(4= 091776K)=0A>>=A0=A0>> > icms_dc=3D6 , 0.0697320 secs] [Times: user=3D0.07 s= ys=3D0.00, =0A> real=3D0.07=0A>>=A0=A0secs]=0A>>=A0=A0>> > 2012-04-30T18:58= :48.447+0000: 105719.555: [GC 105719.555: =0A> [ParNew:=0A>>=A0=A0>> > 3736= 1K->2512K(38336K), 0.0510560 secs] =0A> 3149748K->3116166K(4091776K)=0A>>= =A0=A0>> > icms_dc=3D6 , 0.0512050 secs] [Times: user=3D0.05 sys=3D0.00, = =0A> real=3D0.00=0A>>=A0=A0secs]=0A>>=A0=A0>> > 2012-04-30T18:58:48.794+000= 0: 105719.902: [GC 105719.902: =0A> [ParNew:=0A>>=A0=A0>> > 36624K->2616K(3= 8336K), 0.1986770 secs] =0A> 3150278K->3116270K(4091776K)=0A>>=A0=A0>> > ic= ms_dc=3D6 , 0.1988160 secs] [Times: user=3D0.05 sys=3D0.00, =0A> real=3D0.2= 0=0A>>=A0=A0secs]=0A>>=A0=A0>> > 2012-04-30T18:58:49.614+0000: 105720.722: = [GC 105720.722: =0A> [ParNew:=0A>>=A0=A0>> > 36728K->3450K(38336K), 0.10991= 90 secs] =0A> 3150382K->3117103K(4091776K)=0A>>=A0=A0>> > icms_dc=3D6 , 0.1= 100650 secs] [Times: user=3D0.05 sys=3D0.00, =0A> real=3D0.11=0A>>=A0=A0sec= s]=0A>>=A0=A0>> > 2012-04-30T18:58:50.631+0000: 105721.739: [GC 105721.739:= =0A> [ParNew:=0A>>=A0=A0>> > 37546K->3176K(38336K), 0.2096280 secs] =0A> 3= 151199K->3117154K(4091776K)=0A>>=A0=A0>> > icms_dc=3D6 , 0.2097770 secs] [T= imes: user=3D0.10 sys=3D0.00, =0A> real=3D0.21=0A>>=A0=A0secs]=0A>>=A0=A0>>= > 2012-04-30T18:58:51.300+0000: 105722.408: [GC 105722.408: =0A> [ParNew:= =0A>>=A0=A0>> > 37288K->3928K(38336K), 0.0873820 secs] =0A> 3151266K->31179= 07K(4091776K)=0A>>=A0=A0>> > icms_dc=3D6 , 0.0875310 secs] [Times: user=3D0= .05 sys=3D0.00, =0A> real=3D0.08=0A>>=A0=A0secs]=0A>>=A0=A0>> > 2012-04-30T= 18:58:52.109+0000: 105723.217: [GC 105723.217: =0A> [ParNew:=0A>>=A0=A0>> >= 38040K->2095K(38336K), 0.0791560 secs] =0A> 3152019K->3116839K(4091776K)= =0A>>=A0=A0>> > icms_dc=3D6 , 0.0792970 secs] [Times: user=3D0.05 sys=3D0.0= 0, =0A> real=3D0.08=0A>>=A0=A0secs]=0A>>=A0=A0>> > 2012-04-30T18:58:53.132+= 0000: 105724.240: [GC 105724.240: =0A> [ParNew:=0A>>=A0=A0>> > 36207K->2581= K(38336K), 0.1651210 secs] =0A> 3150951K->3117700K(4091776K)=0A>>=A0=A0>> >= icms_dc=3D6 , 0.1652650 secs] [Times: user=3D0.06 sys=3D0.00, =0A> real=3D= 0.16=0A>>=A0=A0secs]=0A>>=A0=A0>> > 2012-04-30T18:58:55.402+0000: 105726.51= 0: [GC 105726.510: =0A> [ParNew:=0A>>=A0=A0>> > 36693K->2297K(38336K), 0.12= 62480 secs] =0A> 3151812K->3117742K(4091776K)=0A>>=A0=A0>> > icms_dc=3D6 , = 0.1264420 secs] [Times: user=3D0.06 sys=3D0.00, =0A> real=3D0.13=0A>>=A0=A0= secs]=0A>>=A0=A0>> > 2012-04-30T18:58:56.103+0000: 105727.211: [GC 105727.2= 11: =0A> [ParNew:=0A>>=A0=A0>> > 36409K->4224K(38336K), 0.3658760 secs] =0A= > 3151854K->3120205K(4091776K)=0A>>=A0=A0>> > icms_dc=3D6 , 0.3660220 secs]= =0A>>=A0=A0>> >=0A>>=A0=A0>>=0A>> =0A>> =0A>