hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-7224) Allow reuse of NN connections via webhdfs
Date Fri, 10 Oct 2014 13:31:34 GMT
Eric Payne created HDFS-7224:

             Summary: Allow reuse of NN connections via webhdfs
                 Key: HDFS-7224
                 URL: https://issues.apache.org/jira/browse/HDFS-7224
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: webhdfs
    Affects Versions: 2.5.0
            Reporter: Eric Payne
            Assignee: Eric Payne

In very large clusters, the webhdfs client could get bind exceptions because it runs out of
ports. This could happen when using webhdfs to talk to the NN in order to do list globbing
of a
huge amount of files.

WebHdfsFileSystem#jsonParse gets the input/error stream from the connection,
but never closes the stream. Since it's not closed, the JVM thinks the stream may still
be transferring data, so the next time through this code, it has to get a new connection
rather than reusing an existing one. 

The lack of connection reuse has poor latency and adds too much overhead to the NN.

This message was sent by Atlassian JIRA

View raw message