hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7279) Use netty to implement DatanodeWebHdfsMethods
Date Wed, 05 Nov 2014 23:50:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199400#comment-14199400
] 

Haohui Mai commented on HDFS-7279:
----------------------------------

I tested the patch in the following setting. Each DN serves 1k concurrent writers, each of
which writes 4GB in total in 64 separate requests.

The old DN frequently ran into GC and timeouts. I no longer observe these issues after this
patch.

> Use netty to implement DatanodeWebHdfsMethods
> ---------------------------------------------
>
>                 Key: HDFS-7279
>                 URL: https://issues.apache.org/jira/browse/HDFS-7279
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode, webhdfs
>            Reporter: Haohui Mai
>            Assignee: Haohui Mai
>         Attachments: HDFS-7279.000.patch, HDFS-7279.001.patch, HDFS-7279.002.patch, HDFS-7279.003.patch,
HDFS-7279.004.patch, HDFS-7279.005.patch, HDFS-7279.006.patch, HDFS-7279.007.patch
>
>
> Currently the DN implements all related webhdfs functionality using jetty. As the current
jetty version the DN used (jetty 6) lacks of fine-grained buffer and connection management,
DN often suffers from long latency and OOM when its webhdfs component is under sustained heavy
load.
> This jira proposes to implement the webhdfs component in DN using netty, which can be
more efficient and allow more finer-grain controls on webhdfs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message