hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei Guo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3926) Extend the YARN resource model for easier resource-type management and profiles
Date Tue, 30 Aug 2016 14:37:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449183#comment-15449183

Lei Guo commented on YARN-3926:

[~vvasudev],  some thoughts I just commented in Yarn-4793.

As this Jira targets to build a unified interface, I'd like to share some thoughts related
to the resource modeling part. The core of Yarn is still to map the resource and workload.
For the resource modeling proposed in YARN-3926, it extends the current Yarn static resource
modeling to be a flat resource modeling. The end user has the potential to define/schedule
their own resource. I am considering whether we should do further extension to make the resource
modeling to be a hierarchy based modeling. The use case I see for future is the heterogenous
environment with different hardware accelerators (GPU, Intel Xeon Phi, FPGA, etc). For example,
if you treat one GPU as a unit of special resource, the flat resource modeling is good enough.
But we are seeing cases that GPU to be shared between applications, even the application prefer
to allocate certain range of memory inside GPU to avoid cache rotation issue. In this case,
it's hard for scheduler to handle. There is relationship between resource (just like the relationship
between applications in Slider). Scheduler must allocate GPU memory and GPU core on the same

If we do have vision to cover more complicate environments with Yarn, maybe it's time to consider
further extension on the resource modeling together with Slider integration and unified service

> Extend the YARN resource model for easier resource-type management and profiles
> -------------------------------------------------------------------------------
>                 Key: YARN-3926
>                 URL: https://issues.apache.org/jira/browse/YARN-3926
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, resourcemanager
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: Proposal for modifying resource model and profiles.pdf
> Currently, there are efforts to add support for various resource-types such as disk(YARN-2139),
network(YARN-2140), and  HDFS bandwidth(YARN-2681). These efforts all aim to add support for
a new resource type and are fairly involved efforts. In addition, once support is added, it
becomes harder for users to specify the resources they need. All existing jobs have to be
modified, or have to use the minimum allocation.
> This ticket is a proposal to extend the YARN resource model to a more flexible model
which makes it easier to support additional resource-types. It also considers the related
aspect of “resource profiles” which allow users to easily specify the various resources
they need for any given container.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message