hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Zhuge (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11280) Allow WebHDFS to reuse HTTP connections to NN
Date Mon, 02 Jan 2017 22:11:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15793588#comment-15793588

John Zhuge commented on HDFS-11280:

These unit test failures may be caused by this patch: TestWebHDFSXAttr, TestWebHDFS, TestWebHdfsTokens,
TestWebHdfsWithRestCsrfPreventionFilter, and TestAuditLogs. After reverting the trunk to "165d01a
YARN-5931. Document timeout interfaces CLI and REST APIs (Contributed by Rohith Sharma K S
via Daniel Templeton)", these unit tests all passed.


> Allow WebHDFS to reuse HTTP connections to NN
> ---------------------------------------------
>                 Key: HDFS-11280
>                 URL: https://issues.apache.org/jira/browse/HDFS-11280
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 2.7.3, 2.6.5, 3.0.0-alpha1
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>             Fix For: 2.8.0, 2.9.0, 2.7.4, 3.0.0-alpha2
>         Attachments: HDFS-11280.for.2.7.and.below.patch, HDFS-11280.for.2.8.and.beyond.2.patch,
HDFS-11280.for.2.8.and.beyond.3.patch, HDFS-11280.for.2.8.and.beyond.4.patch, HDFS-11280.for.2.8.and.beyond.patch
> WebHDFSClient calls "conn.disconnect()", which disconnects from the NameNode.  When we
use webhdfs as the source in distcp, this used up all ephemeral ports on the client side since
all closed connections continue to occupy the port with TIME_WAIT status for some time.
> According to http://tinyurl.com/java7-http-keepalive, we should call conn.getInputStream().close()
instead to make sure the connection is kept alive.  This will get rid of the ephemeral port
> Manual steps used to verify the bug fix:
> 1. Build original hadoop jar.
> 2. Try out distcp from webhdfs as source, and "netstat -n | grep TIME_WAIT | grep -c
50070" on the local machine shows a big number (100s).
> 3. Build hadoop jar with this diff.
> 4. Try out distcp from webhdfs as source, and "netstat -n | grep TIME_WAIT | grep -c
50070" on the local machine shows 0.
> 5. The explanation:  distcp's client side does a lot of directory scanning, which would
create and close a lot of connections to the namenode HTTP port.
> Reference:
> 2.7 and below: https://github.com/apache/hadoop/blob/branch-2.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L743
> 2.8 and above: https://github.com/apache/hadoop/blob/branch-2.8/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L898

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message