hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: taking snapshot's creates to many TCP CLOSE_WAIT handles on the hbase master server
Date Sun, 20 Apr 2014 00:05:12 GMT
On Thu, Apr 17, 2014 at 9:50 PM, Stack <stack@duboce.net> wrote:

> On Thu, Apr 17, 2014 at 6:51 AM, Hansi Klose <hansi.klose@web.de> wrote:
>> Hi,
>> we use a script to take on a regular basis snapshot's and delete old
>> one's.
>> We recognizes that the web interface of the hbase master was not working
>> any more becaues of too many open files.
>> The master reaches his number of open file limit of 32768
>> When I run lsof I saw that there where a lot of TCP CLOSE_WAIT handles
>> open
>> with the regionserver as target.
>> On the regionserver there is just one connection to the hbase master.
>> I can see that the count of the CLOSE_WAIT handles grow each time
>> i take a snapshot. When i delete on nothing changes.
>> Each time i take a snapshot  there are 20 - 30 new CLOSE_WAIT handles.
>> Why does the master do not close the handles? Is there a parameter
>> with a timeout we can use?
>> We use hbase 0.94.2-cdh4.2.0.
> Does
> https://issues.apache.org/jira/browse/HBASE-9393?jql=text%20~%20%22CLOSE_WAIT%22help?
 In particular, what happens if you up the socket cache as suggested
> on the end of the issue?
> HDFS-4911 may help (the CLOSE_WAIT is against local/remote DN, right?) or
quoting one of our lads off an internal issue, "You could get most of the
same benefit of HDFS-4911...by setting dfs.client.socketcache.expiryMsec to
900 in your HBase client configuration. The goal is that the client should
not hang on to sockets longer than the DataNode does...."

Or, can you upgrade?



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message