hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jack Levin <magn...@gmail.com>
Subject more on topic of REST getting frozen while PUTting
Date Thu, 23 Dec 2010 19:01:35 GMT
I found this in our gc.log:


2010-12-23T10:48:48.390-0800: 1172863.477: [Full GC [PSYoungGen:
5156K->0K(22720K)] [PSOldGen: 1296647K->1172950K(1278336K)]
1301804K->1172950K(1301056K) [PSPermGen: 18458K->18458K(21248K)],
0.3994190 secs] [Times: user=0.39 sys=0.00, real=0.39 secs]
2010-12-23T10:48:51.656-0800: 1172866.744: [GC [PSYoungGen:
15003K->5348K(20352K)] 1187954K->1178299K(1298688K), 0.0199330 secs]
[Times: user=0.04 sys=0.00, real=0.02 secs]
2010-12-23T10:48:56.170-0800: 1172871.258: [GC [PSYoungGen:
20322K->4964K(22208K)] 1193272K->1183103K(1300544K), 0.0183390 secs]
[Times: user=0.04 sys=0.00, real=0.01 secs]
2010-12-23T10:48:59.667-0800: 1172874.755: [GC [PSYoungGen:
19876K->5316K(20224K)] 1198015K->1188308K(1298560K), 0.0195830 secs]
[Times: user=0.05 sys=0.00, real=0.02 secs]
2010-12-23T10:49:03.458-0800: 1172878.545: [GC [PSYoungGen:
20126K->5188K(21696K)] 1203117K->1193289K(1300032K), 0.0107620 secs]
[Times: user=0.04 sys=0.00, real=0.01 secs]
2010-12-23T10:49:07.646-0800: 1172882.733: [GC [PSYoungGen:
19945K->5123K(19904K)] 1208045K->1198212K(1298240K), 0.0133460 secs]
[Times: user=0.05 sys=0.00, real=0.01 secs]
2010-12-23T10:49:12.130-0800: 1172887.217: [GC [PSYoungGen:
19843K->6949K(20096K)] 1212932K->1203016K(1298432K), 0.0148730 secs]
[Times: user=0.06 sys=0.00, real=0.02 secs]
2010-12-23T10:49:15.450-0800: 1172890.537: [GC [PSYoungGen:
20053K->4612K(17728K)] 1216120K->1207573K(1296064K), 0.0205570 secs]
[Times: user=0.05 sys=0.00, real=0.02 secs]
2010-12-23T10:49:18.362-0800: 1172893.449: [GC [PSYoungGen:
17668K->4643K(21120K)] 1220629K->1212008K(1299456K), 0.0137250 secs]
[Times: user=0.05 sys=0.00, real=0.02 secs]
2010-12-23T10:49:21.081-0800: 1172896.169: [GC [PSYoungGen:
17635K->4419K(17408K)] 1225000K->1216267K(1295744K), 0.0141270 secs]
[Times: user=0.05 sys=0.00, real=0.02 secs]
2010-12-23T10:49:23.948-0800: 1172899.035: [GC [PSYoungGen:
17342K->4548K(20736K)] 1229190K->1220680K(1299072K), 0.0109070 secs]
[Times: user=0.04 sys=0.00, real=0.01 secs]
2010-12-23T10:49:27.450-0800: 1172902.537: [GC [PSYoungGen:
17412K->4611K(17472K)] 1233544K->1225076K(1295808K), 0.0134810 secs]
[Times: user=0.05 sys=0.00, real=0.01 secs]
2010-12-23T10:49:30.224-0800: 1172905.311: [GC [PSYoungGen:
17356K->4419K(20352K)] 1237820K->1229231K(1298688K), 0.0128090 secs]
[Times: user=0.05 sys=0.00, real=0.02 secs]
2010-12-23T10:49:33.649-0800: 1172908.736: [GC [PSYoungGen:
17150K->4515K(17216K)] 1241962K->1233595K(1295552K), 0.0170930 secs]
[Times: user=0.04 sys=0.01, real=0.02 secs]
2010-12-23T10:49:36.906-0800: 1172911.993: [GC [PSYoungGen:
17176K->4355K(19904K)] 1246256K->1237775K(1298240K), 0.0098390 secs]
[Times: user=0.04 sys=0.00, real=0.01 secs]
2010-12-23T10:49:40.824-0800: 1172915.911: [GC [PSYoungGen:
16963K->7462K(20032K)] 1250383K->1241898K(1298368K), 0.0174860 secs]
[Times: user=0.06 sys=0.00, real=0.02 secs]
2010-12-23T10:49:43.492-0800: 1172918.579: [GC [PSYoungGen:
20002K->4483K(20672K)] 1254438K->1246310K(1299008K), 0.0195380 secs]
[Times: user=0.06 sys=0.00, real=0.02 secs]
2010-12-23T10:49:46.529-0800: 1172921.616: [GC [PSYoungGen:
16963K->4452K(16896K)] 1258790K->1250562K(1295232K), 0.0181960 secs]
[Times: user=0.06 sys=0.00, real=0.01 secs]
2010-12-23T10:49:49.648-0800: 1172924.735: [GC [PSYoungGen:
16819K->4355K(20224K)] 1262929K->1254766K(1298560K), 0.0116060 secs]
[Times: user=0.03 sys=0.00, real=0.02 secs]
2010-12-23T10:49:52.448-0800: 1172927.535: [GC [PSYoungGen:
16686K->4419K(16768K)] 1267097K->1258977K(1295104K), 0.0166160 secs]
[Times: user=0.06 sys=0.00, real=0.02 secs]
2010-12-23T10:49:56.601-0800: 1172931.688: [GC [PSYoungGen:
16657K->4099K(19776K)] 1271215K->1262933K(1298112K), 0.0145490 secs]
[Times: user=0.06 sys=0.00, real=0.01 secs]
2010-12-23T10:49:59.592-0800: 1172934.679: [GC [PSYoungGen:
16274K->4227K(16448K)] 1275107K->1267024K(1294784K), 0.0188570 secs]
[Times: user=0.05 sys=0.00, real=0.02 secs]
2010-12-23T10:50:03.008-0800: 1172938.095: [GC [PSYoungGen:
16380K->4291K(19264K)] 1279177K->1271099K(1297600K), 0.0202760 secs]
[Times: user=0.06 sys=0.00, real=0.02 secs]
2010-12-23T10:50:06.085-0800: 1172941.172: [GC [PSYoungGen:
16387K->4291K(16384K)] 1283195K->1275247K(1294720K), 0.0127600 secs]
[Times: user=0.05 sys=0.00, real=0.01 secs]
Heap
 PSYoungGen      total 16384K, used 14153K [0x00007f9493560000,
0x00007f9494f40000, 0x00007f94bd000000)
  eden space 12032K, 81% used
[0x00007f9493560000,0x00007f9493f01820,0x00007f9494120000)
  from space 4352K, 98% used
[0x00007f9494130000,0x00007f9494560e20,0x00007f9494570000)
  to   space 7232K, 0% used
[0x00007f9494830000,0x00007f9494830000,0x00007f9494f40000)
 PSOldGen        total 1278336K, used 1270955K [0x00007f9440000000,
0x00007f948e060000, 0x00007f9493560000)
  object space 1278336K, 99% used
[0x00007f9440000000,0x00007f948d92aff0,0x00007f948e060000)
 PSPermGen       total 21248K, used 18487K [0x00007f943ac00000,
0x00007f943c0c0000, 0x00007f9440000000)
  object space 21248K, 87% used
[0x00007f943ac00000,0x00007f943be0dc58,0x00007f943c0c0000)


process:

root     12779 10.0  2.3 8925136 377464 ?      Sl   06:41  25:20
/usr/java/jdk1.6.0_12/bin/java -Xmx6000m -XX:+UseConcMarkSweepGC
-XX:MaxDirectMemorySize=6G -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -Xloggc:/usr/lib/hbase/bin/../logs/gc-hbase.log
-XX:+UseConcMarkSweepGC -XX:MaxDirectMemorySize=6G -verbose:gc
-XX:+PrintGCDetails


jstack is here:
http://pastebin.com/eW6ReKnb

and the rest is here: http://pastebin.com/1p6EQgqS

Now, not PUTs or GETs can come thru

-Jack

Mime
View raw message