Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Tue, 31 Dec 2013 10:13:50 +0000 (UTC)
From: "Uma Maheswara Rao G (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12684916.1387188598544.26090.1388484830570@arcas>
In-Reply-To: <JIRA.12684916.1387188598544@arcas>
References: <JIRA.12684916.1387188598544@arcas>
Subject: [jira] [Commented] (HDFS-5671) When client  request block to
 DataNode and "java.io.IOException" occurs, the fail TCP socket is not
 closed (in status "CLOSE_WAIT" with port 1004 of DataNode)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859426#comment-13859426 ] 

Uma Maheswara Rao G commented on HDFS-5671:
-------------------------------------------

I am able to apply this patch locally to trunk. Not sure why its failing in jenkins. Does anyone has idea on this?


> When client  request block to DataNode and "java.io.IOException" occurs, the fail TCP socket is not closed (in status "CLOSE_WAIT" with port 1004 of DataNode)
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-5671
>                 URL: https://issues.apache.org/jira/browse/HDFS-5671
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 2.2.0
>         Environment: hadoop-2.2.0
> java version "1.6.0_31"
> Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode)
> Linux 2.6.32-358.14.1.el6.x86_64 #1 SMP Tue Jul 16 23:51:20 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>            Reporter: JamesLi
>            Priority: Critical
>         Attachments: 5671.patch, 5671v1.patch, 5671v2.patch, 5671v3.patch, 5671v4.patch
>
>
> lsof -i TCP:1004 | grep -c CLOSE_WAIT
> 18235
> When client request a file's block to DataNode:1004. If request fail because "java.io.IOException: Got error for OP_READ_BLOCK,Block token is expired." Occurs  and the TCP socket that regionserver using is not closed.
> I think the problem above is in DatanodeInfo blockSeekTo(long target)  of Class DFSInputStream 
> The connection client using is BlockReader: 
>         blockReader = getBlockReader(targetAddr, chosenNode, src, blk,
>             accessToken, offsetIntoBlock, blk.getNumBytes() - offsetIntoBlock,
>             buffersize, verifyChecksum, dfsClient.clientName);
> In DFSInputStream.blockSeekTo()-line 533,invoke getBlockReader() which wil generate a peer use newTcpPeer(dnAddr) -line 1107,when BlockReaderFactory.newBlockReader throw IOException,the peer will not be closed which will cause a CLOSE_WAIT connection. 
> In our test,when datanode get a InvalidToken exception in DataXceiver.checkAccess(),it will close the connection.At regionserver side, in RemoteBlockReader2.newBlockReader(),checkSuccess() will throw a InvalidBlockTokenException, DFSInputStream.blockSeekTo() will catch the exception, but the connection is NOT closed, it become CLOSE_WAIT.


--
This message was sent by Atlassian JIRA
(v6.1.5#6160)