hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bing Jiang <jiangbinglo...@gmail.com>
Subject Too Many CLOSE_WAIT make performance down
Date Wed, 07 Aug 2013 06:03:49 GMT
Version:
HBase: 0.94.3
HDFS: 0.20.*

There are too many CLOSE_WAIT connection from RS to DN, and I find the
number is over 30000.
And change the Log-Level of 'org.apache.hadoop.ipc.HBaseServer.trace' to
DEBUG, and check that the performance:

> Call #2649932; Served: HRegionInterface#get queueTime=0 processingTime=284
> contents=1 Get, 86 bytes
>
> So the conclusion is that when DataNode server port has been occupied by
normal or  irregular connection, it will bring read/write performance down.

According to principle of TCP/IP protocol, CLOSE_WAIT means that RS cannot
close fd which has been opened, and I restart RS gracefully, the problem
has been tackled. Ok, My question is :

Can someone tell me in which conditions do RS will ignore the file handler?

Any ideas will be nice.

Thanks!

-- 
Bing Jiang
Tel:(86)134-2619-1361
weibo: http://weibo.com/jiangbinglover
BLOG: www.binospace.com
BLOG: http://blog.sina.com.cn/jiangbinglover
Focus on distributed computing, HDFS/HBase

Mime
View raw message