hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng Shao (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-11280) Allow WebHDFSClient to reuse HTTP connections (HTTP Keep-Alive)
Date Thu, 29 Dec 2016 01:54:58 GMT
Zheng Shao created HDFS-11280:

             Summary: Allow WebHDFSClient to reuse HTTP connections (HTTP Keep-Alive)
                 Key: HDFS-11280
                 URL: https://issues.apache.org/jira/browse/HDFS-11280
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: hdfs
    Affects Versions: 3.0.0-alpha1, 2.6.5, 2.7.3
            Reporter: Zheng Shao
            Priority: Minor

WebHDFSClient calls "conn.disconnect()", which disconnects from the NameNode.  When we use
webhdfs as the source in distcp, this used up all ephemeral ports since all closed connections
continue to occupy the port with TIME_WAIT status for some time.

According to tinyurl.com/java7-http-keepalive, we should call conn.getInputStream().close()
instead to make sure the connection is kept alive.  This will get rid of the ephemeral port

2.7 and below: https://github.com/apache/hadoop/blob/branch-2.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L743

2.8 and above: https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L898

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message