hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7607) Use random rack-local node for webhdfs opens to avoid OOM on DNs
Date Thu, 12 May 2016 22:21:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15282155#comment-15282155

Sangjin Lee commented on HDFS-7607:

There has been no movement on this for a while. We'll need to move it out of scope for 2.8.0
soon. Let me know if you disagree.

> Use random rack-local node for webhdfs opens to avoid OOM on DNs
> ----------------------------------------------------------------
>                 Key: HDFS-7607
>                 URL: https://issues.apache.org/jira/browse/HDFS-7607
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode, webhdfs
>    Affects Versions: 2.0.0-alpha
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Critical
> Webhdfs currently redirects a client to the DN that physically has one of the
> replicas.  Unlike the hdfs data streamer protocol which can easily handle hundreds or
thousands of connections, jetty has poor performance under heavy load.  Webhdfs clients can
easily overwhelm the DNs and likely cause OOMs or excessive GC.
> The NN should redirect the client to a rack-local location to distribute the webhdfs
load across multiple hosts.  The rack can then use the lightweight streamer protocol.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message