hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laxman <lakshman...@huawei.com>
Subject RE: Hmaster and HRegionServer disappearance reason to ask
Date Wed, 11 Jul 2012 04:41:11 GMT
IIUC, this seems to be a problem with Direct memory cleanup.
I've seen this problem mostly with application using lot of NIO
(RegionServer, DataNode).

Symptoms of this problem
1) Allocated heap is 1 GB
2) VIRT/RES will occupy 10GB or even more
3) GC logs says, it takes more than 1 minute to clean up where allocated
heap is just few GB (<4GB)
4) Your application uses lot of NIO/RMI calls(DataNode, RegionServer)

How to solve?
1) Fix the direct memory usage to a fixed value -XX:MaxDirectMemorySize=1G
2)  DON'T set -XX:+DisableExplicitGC

Recent issue which is relevant to this HDFS-3600.

Please let us know your findings.
--
Regards,
Laxman


> -----Original Message-----
> From: Pablo Musa [mailto:pablo@psafe.com]
> Sent: Tuesday, July 10, 2012 7:58 PM
> To: user@hbase.apache.org
> Subject: RE: Hmaster and HRegionServer disappearance reason to ask
> 
> I tried to change the flag but yesterday it happened again:
> 
> Application time: 0.3025790 seconds
> 30013.866: [GC 30013.866: [ParNew: 106069K->989K(118016K), 178.8437590
> secs] 473853K->369013K(1275392K), 178.8438570 secs] [Times: user=0.05
> sys=178.82, real=178.81 secs]
> Total time for which application threads were stopped: 178.8441500
> seconds
> 
> I also checked the possibility of a swap, but I don't think it is the
> problem as vmstat always show clean swap.
> 
> Help guys, please :)
> 
> Abs,
> Pablo



Mime
View raw message