hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng Shao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-11280) Allow WebHDFSClient to reuse HTTP connections (HTTP Keep-Alive)
Date Fri, 30 Dec 2016 18:59:59 GMT

     [ https://issues.apache.org/jira/browse/HDFS-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zheng Shao updated HDFS-11280:
------------------------------
    Description: 
WebHDFSClient calls "conn.disconnect()", which disconnects from the NameNode.  When we use
webhdfs as the source in distcp, this used up all ephemeral ports on the client side since
all closed connections continue to occupy the port with TIME_WAIT status for some time.

According to http://tinyurl.com/java7-http-keepalive, we should call conn.getInputStream().close()
instead to make sure the connection is kept alive.  This will get rid of the ephemeral port
problem.

Reference:
2.7 and below: https://github.com/apache/hadoop/blob/branch-2.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L743

2.8 and above: https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L898



  was:
WebHDFSClient calls "conn.disconnect()", which disconnects from the NameNode.  When we use
webhdfs as the source in distcp, this used up all ephemeral ports since all closed connections
continue to occupy the port with TIME_WAIT status for some time.

According to http://tinyurl.com/java7-http-keepalive, we should call conn.getInputStream().close()
instead to make sure the connection is kept alive.  This will get rid of the ephemeral port
problem.

Reference:
2.7 and below: https://github.com/apache/hadoop/blob/branch-2.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L743

2.8 and above: https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L898




> Allow WebHDFSClient to reuse HTTP connections (HTTP Keep-Alive)
> ---------------------------------------------------------------
>
>                 Key: HDFS-11280
>                 URL: https://issues.apache.org/jira/browse/HDFS-11280
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 2.7.3, 2.6.5, 3.0.0-alpha1
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>         Attachments: HDFS-11280.for.2.7.and.below.patch, HDFS-11280.for.2.8.and.beyond.2.patch,
HDFS-11280.for.2.8.and.beyond.patch
>
>
> WebHDFSClient calls "conn.disconnect()", which disconnects from the NameNode.  When we
use webhdfs as the source in distcp, this used up all ephemeral ports on the client side since
all closed connections continue to occupy the port with TIME_WAIT status for some time.
> According to http://tinyurl.com/java7-http-keepalive, we should call conn.getInputStream().close()
instead to make sure the connection is kept alive.  This will get rid of the ephemeral port
problem.
> Reference:
> 2.7 and below: https://github.com/apache/hadoop/blob/branch-2.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L743
> 2.8 and above: https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L898



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message