hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5671) When Hbase RegionServer request block to DataNode and "java.io.IOException" occurs, the fail TCP socket is not closed (in status "CLOSE_WAIT" with port 1004 of DataNode)
Date Tue, 17 Dec 2013 04:39:08 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13850100#comment-13850100
] 

stack commented on HDFS-5671:
-----------------------------

Would you have a patch [~jamesli]?  If so, would you mind posting?  Could you try it on your
cluster to see if it cuts down on instances of CLOSE_WAIT?  Thanks.

> When Hbase RegionServer request block to DataNode and "java.io.IOException" occurs, the
fail TCP socket is not closed (in status "CLOSE_WAIT" with port 1004 of DataNode)
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-5671
>                 URL: https://issues.apache.org/jira/browse/HDFS-5671
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 2.2.0
>         Environment: hadoop-2.2.0
> java version "1.6.0_31"
> Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
> Linux 2.6.32-358.14.1.el6.x86_64 #1 SMP Tue Jul 16 23:51:20 UTC 2013 x86_64 x86_64 x86_64
GNU/Linux
>            Reporter: JamesLi
>            Priority: Critical
>
> lsof -i TCP:1004 | grep -c CLOSE_WAIT
> 18235
> When hbase regionserver request a file's block to DataNode:1004. If request fail because
"java.io.IOException: Got error for OP_READ_BLOCK,Block token is expired." Occurs  and the
TCP socket that regionserver using is not closed.
> I think the problem above is in DatanodeInfo blockSeekTo(long target) - line 539 of Class
DFSInputStream 
> The connection regionserver using is BlockReader, it created on line 574 : 
>         blockReader = getBlockReader(targetAddr, chosenNode, src, blk,
>             accessToken, offsetIntoBlock, blk.getNumBytes() - offsetIntoBlock,
>             buffersize, verifyChecksum, dfsClient.clientName);
> and if this connection fail, regionserver will fetch a new access token , and old Connection
is not closed here. 
> I think need small code to close old Connection when exception happens:
>     	if(blockReader != null)
>     		try{
>     			blockReader.close();
>     			blockReader = null;
>     		} catch (IOException exc) {
>     			DFSClient.LOG.error("Close connection to " + targetAddr + " failed");
>     		} 



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message