Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of kuosenhao@gmail.com
 designates 209.85.213.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=pM7ehAiTs6ZtaQyOaezwx6Z2jZ1ndDj8//sBHbipSF8LpjYfHm5YxUT+99jCgAwYPe
         ww1CYKGkGpEy8hRxdg+WpZ19CNmLfRJ+lDnxccsx2Mp19K/nFw5ATNaZt/86XxxkEDgj
         GtDOcgMmP8dPrh+LiOUS1rJUOr4FIlAakncYQ=
MIME-Version: 1.0
Date: Thu, 29 Jul 2010 13:17:07 -0700
Message-ID: <AANLkTin_gibY+aUUnfhRvihne0C7Da7NA4OTh7EtiSA=@mail.gmail.com>
Subject: GC [ParNew...] took 299 secs causing region server to die
From: Steve Kuo <kuosenhao@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=0016368e258f24a4d9048c8c6b3b

--0016368e258f24a4d9048c8c6b3b
Content-Type: text/plain; charset=ISO-8859-1

I kept running into the stop-the-world GC during batch import of data into
hbase.  The configuration of a node in the 8-node cluster is as follows.

* 4-core
* 64-bit JVM
* 8 GB of memory
* CDH2 for hadoop and 0.20.5 for hbase
* TT: 128 MB
* DN: 128 MB
* 2 Mappers at 512 MB each
* 2 Reducer at 512 MB each
* 1 regionserver at 4096 MB

The import job was a mapper only job so that only TT, DN, 2 mappers and
regionserver were running.  Below is the JMX output for the dead
regionserver.

Time:
2010-07-29 12:25:47
Used:
   224,949 kbytes
Committed:
   670,728 kbytes
Max:
 4,185,792 kbytes
GC time:
     5 minutes on ParNew (2,126 collections)
 0.000 seconds on ConcurrentMarkSweep (0 collections)

Clearly the regionserver was spent all GC time on ParNew, which was not
surprising as I was imported tons of data.  But I could not figure out why
the same GC that usually take way less than a second, took 299 secs at line
3.  Any enlightenment is greatly appreciated.

I will change ParNew to 6M as documented in Performance Tuning page and gave
it another shot.

010-07-28T12:06:57.249-0700: 2406.986: [GC 2406.986: [ParNew:
17786K->755K(19136K), 0.0015410 secs] 348288K->331394K(620416K) icms_dc=27 ,
0.0016330 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-07-28T12:06:57.268-0700: 2407.004: [GC 2407.004: [ParNew:
17580K->761K(19136K), 0.0016710 secs] 348154K->331343K(620416K) icms_dc=27 ,
0.0017610 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2010-07-28T12:06:57.288-0700: 2407.024: [GC 2407.088: [ParNew:
17564K->757K(19136K), 299.1513910 secs] 348081K->331283K(620416K) icms_dc=27
, 299.1515120 secs] [Times: user=0.17 sys=0.04, real=299.23 secs]
2010-07-28T12:11:56.558-0700: 2706.294: [GC 2706.294: [ParNew:
17735K->925K(19136K), 0.0094600 secs] 348197K->331458K(620416K) icms_dc=27 ,
0.0095670 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
2010-07-28T12:11:56.606-0700: 2706.343: [GC 2706.343: [ParNew:
17940K->932K(19136K), 0.0085750 secs] 348473K->331474K(620416K) icms_dc=27 ,
0.0086710 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]

--0016368e258f24a4d9048c8c6b3b--