hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7607) Use random rack-local node for webhdfs opens to avoid OOM on DNs
Date Wed, 14 Jan 2015 15:58:37 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277110#comment-14277110

Daryn Sharp commented on HDFS-7607:

Yes, but it will probably take a few weeks.  Maybe we should retarget for 2.6.1 in case that
release is made?  We've been running with this patch since last fall otherwise webhdfs would
cause cluster meltdowns, so others may find it useful.

> Use random rack-local node for webhdfs opens to avoid OOM on DNs
> ----------------------------------------------------------------
>                 Key: HDFS-7607
>                 URL: https://issues.apache.org/jira/browse/HDFS-7607
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode, webhdfs
>    Affects Versions: 2.0.0-alpha
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Critical
> Webhdfs currently redirects a client to the DN that physically has one of the
> replicas.  Unlike the hdfs data streamer protocol which can easily handle hundreds or
thousands of connections, jetty has poor performance under heavy load.  Webhdfs clients can
easily overwhelm the DNs and likely cause OOMs or excessive GC.
> The NN should redirect the client to a rack-local location to distribute the webhdfs
load across multiple hosts.  The rack can then use the lightweight streamer protocol.

This message was sent by Atlassian JIRA

View raw message