hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7279) Use netty to implement DatanodeWebHdfsMethods
Date Thu, 23 Oct 2014 00:17:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180784#comment-14180784
] 

Haohui Mai commented on HDFS-7279:
----------------------------------

An alternative option is to upgrade jetty and servlet. The New APIs from both jetty and servlet
such as asynchronous servlet can amend some of the issues. Webhdfs on the DN side, however,
is data intensive which does not fit the servlet API very well. The servlet / jetty APIs do
not give fine-grain control on the resources that netty is able to provide. These controls
are critical if webhdfs needs to survive on heavy workload.

The strategy is proven by the mapreduce client, which already uses netty to implement the
shuffle functionality. For other URLs on the DNs, I plan to keep jetty listening on a local
address, but to have a reverse proxy in netty to continue the serve these URLs.

> Use netty to implement DatanodeWebHdfsMethods
> ---------------------------------------------
>
>                 Key: HDFS-7279
>                 URL: https://issues.apache.org/jira/browse/HDFS-7279
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Haohui Mai
>            Assignee: Haohui Mai
>
> Currently the DN implements all related webhdfs functionality using jetty. As the current
jetty version the DN used (jetty 6) lacks of fine-grained buffer and connection management,
DN often suffers from long latency and OOM when its webhdfs component is under sustained heavy
load.
> This jira proposes to implement the webhdfs component in DN using netty, which can be
more efficient and allow more finer-grain controls on webhdfs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message