hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7292) Revisit Resource Profile Behavior
Date Tue, 13 Feb 2018 04:43:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361814#comment-16361814

Wangda Tan commented on YARN-7292:


Let me explain a bit:
- Resource profile now can be specified in ContainerRequest:
    public ContainerRequest(Resource capability, String[] nodes, String[] racks,
        Priority priority, long allocationRequestId, boolean relaxLocality,
        String nodeLabelsExpression,
        ExecutionTypeRequest executionTypeRequest, String profile)
The underlying logic is:
Resource actual = Resource.clone(profileMap.get(profile));
for (ResourceInformation info : capability.getResourceInformations()) {
   if (info.getValue() > 0) {
      actual.set(info.getName(), info.getValue());
// Return actual

Do you think we should add a "do-not-override" option to ContainerRequest constructor? Application
developer pass a zero capability to use profile resource, or pass a null profile to use capability.
I felt it should be enough.

Since we completely removed profile from ResourceRequest, that means an application (like
MR) doesn't use AMRMClient can implement their own logic to do override. Do you think should
we put the logic to a Util class like ResourceUtil (Or create a new one like ResourceProfileUtils)?

> Revisit Resource Profile Behavior
> ---------------------------------
>                 Key: YARN-7292
>                 URL: https://issues.apache.org/jira/browse/YARN-7292
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Blocker
>         Attachments: YARN-7292.002.patch, YARN-7292.003.patch, YARN-7292.004.patch, YARN-7292.005.patch,
YARN-7292.006.patch, YARN-7292.wip.001.patch
> Had discussions with [~templedf], [~vvasudev], [~sunilg] offline. There're a couple of
resource profile related behaviors might need to be updated:
> 1) Configure resource profile in server side or client side: 
> Currently resource profile can be only configured centrally:
> - Advantages:
> A given resource profile has a the same meaning in the cluster. It won’t change when
we run different apps in different configurations. A job can run under Amazon’s G2.8X can
also run on YARN with G2.8X profile. A side benefit is YARN scheduler can potentially do better
bin packing.
> - Disadvantages: 
> Hard for applications to add their own resource profiles. 
> 2) Do we really need mandatory resource profiles such as minimum/maximum/default? 
> 3) Should we send resource profile name inside ResourceRequest, or should client/AM translate
it to resource and set it to the existing resource fields? 
> 4) Related to above, should we allow resource overrides or client/AM should send final
resource to RM?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message