hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7607) Use random rack-local node for webhdfs opens to avoid OOM on DNs
Date Tue, 13 Jan 2015 21:57:36 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276026#comment-14276026
] 

Haohui Mai commented on HDFS-7607:
----------------------------------

Can you verify whether HDFS-7279 has fixed your problem?

> Use random rack-local node for webhdfs opens to avoid OOM on DNs
> ----------------------------------------------------------------
>
>                 Key: HDFS-7607
>                 URL: https://issues.apache.org/jira/browse/HDFS-7607
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode, webhdfs
>    Affects Versions: 2.0.0-alpha
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Critical
>
> Webhdfs currently redirects a client to the DN that physically has one of the
> replicas.  Unlike the hdfs data streamer protocol which can easily handle hundreds or
thousands of connections, jetty has poor performance under heavy load.  Webhdfs clients can
easily overwhelm the DNs and likely cause OOMs or excessive GC.
> The NN should redirect the client to a rack-local location to distribute the webhdfs
load across multiple hosts.  The rack can then use the lightweight streamer protocol.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message