hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaobing Zhou (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8855) Webhdfs client leaks active NameNode connections
Date Thu, 20 Aug 2015 01:16:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14704102#comment-14704102
] 

Xiaobing Zhou commented on HDFS-8855:
-------------------------------------

Thanks [~wheat9] and [~bobhansen] for review. Tracing down to ProtobufRpcEngine that calls
Client.getConnection which fetches connection from cache(com.google.common.cache.Cache) by
using ConnectionId as key. ConnectionId is different for every webhdfs request even if the
url and user are same. That's why the NN connection is constantly created by DN in this case.
Need to refactor ConnectionId(hashCode or equals) somehow to meet comparison invariant assumed
by cache(com.google.common.cache.Cache) to make it work properly.

> Webhdfs client leaks active NameNode connections
> ------------------------------------------------
>
>                 Key: HDFS-8855
>                 URL: https://issues.apache.org/jira/browse/HDFS-8855
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: webhdfs
>         Environment: HDP 2.2
>            Reporter: Bob Hansen
>            Assignee: Xiaobing Zhou
>         Attachments: HDFS-8855.1.patch, HDFS_8855.prototype.patch
>
>
> The attached script simulates a process opening ~50 files via webhdfs and performing
random reads.  Note that there are at most 50 concurrent reads, and all webhdfs sessions are
kept open.  Each read is ~64k at a random position.  
> The script periodically (once per second) shells into the NameNode and produces a summary
of the socket states.  For my test cluster with 5 nodes, it took ~30 seconds for the NameNode
to have ~25000 active connections and fails.
> It appears that each request to the webhdfs client is opening a new connection to the
NameNode and keeping it open after the request is complete.  If the process continues to run,
eventually (~30-60 seconds), all of the open connections are closed and the NameNode recovers.
 
> This smells like SoftReference reaping.  Are we using SoftReferences in the webhdfs client
to cache NameNode connections but never re-using them?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message