hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoyu Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13128) Manage Hadoop RPC resource usage via resource coupon
Date Thu, 15 Sep 2016 18:40:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15494185#comment-15494185

Xiaoyu Yao commented on HADOOP-13128:

Create HADOOP-13128 branch for resource token related development.

> Manage Hadoop RPC resource usage via resource coupon
> ----------------------------------------------------
>                 Key: HADOOP-13128
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13128
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Xiaoyu Yao
>            Assignee: Xiaoyu Yao
>         Attachments: HADOOP-13128-Proposal-20160511.pdf
> HADOOP-9640 added RPC Fair Call Queue and HADOOP-10597 added RPC backoff to ensure the
fairness usage of the HDFS namenode resources. YARN, the Hadoop cluster resource manager currently
manages the CPU and Memory resources for jobs/tasks but not the storage resources such as
HDFS namenode and datanode usage directly. As a result of that, a high priority Yarn Job may
send too many RPC requests to HDFS namenode and get demoted into low priority call queues
due to lack of reservation/coordination. 
> To better support multi-tenancy use cases like above, we propose to manage RPC server
resource usage via coupon mechanism integrated with YARN. The idea is to allow YARN request
HDFS storage resource coupon (e.g., namenode RPC calls, datanode I/O bandwidth) from namenode
on behalf of the job upon submission time.  Once granted, the tasks will include the coupon
identifier in RPC header for the subsequent calls. HDFS namenode RPC scheduler maintains the
state of the coupon usage based on the scheduler policy (fairness or priority) to match the
RPC priority with the YARN scheduling priority.
> I will post a proposal with more detail shortly. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message