hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2496) [YARN-796] Changes for capacity scheduler to support allocate resource respect labels
Date Sat, 20 Sep 2014 14:29:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142001#comment-14142001

Wangda Tan commented on YARN-2496:

Craig, still about #2, I think what you commented is make sense to me, AM can get a more precise
headroom to plan its following resource usage, but I think:
1) It may not enough as what you said:
bq. For this reason, headroom should reflect the labels in the last resource request from
the application, not the queue's labels.
It is possible an AM sent resource requests with different label expression, so what we will
response headroom back to AM?
I think maybe we need a new field in AllocateRequest to request different headrooms under
different label expression.

2) Even with 1), I cannot think of a good way to fast computing random label expression in
an acceptable time complexity, it is possible thousands of different label expression existed
in a big cluster at the same time. Our current implementation can make sure resource of labels
of a queue will up-to-date whenever resource change happened.

With 1) and 2). I suggest to make it as a pending task, and we can deal it in the future.

bq. -re 5, I though * could be in requests, if no, then should not be an issue.
Yes, we doesn't support specify * in requests, because it may cause some possible resource
wastage. AM should clearly know what resource it needed.


> [YARN-796] Changes for capacity scheduler to support allocate resource respect labels
> -------------------------------------------------------------------------------------
>                 Key: YARN-2496
>                 URL: https://issues.apache.org/jira/browse/YARN-2496
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch
> This JIRA Includes:
> - Add/parse labels option to {{capacity-scheduler.xml}} similar to other options of queue
like capacity/maximum-capacity, etc.
> - Include a "default-label-expression" option in queue config, if an app doesn't specify
label-expression, "default-label-expression" of queue will be used.
> - Check if labels can be accessed by the queue when submit an app with labels-expression
to queue or update ResourceRequest with label-expression
> - Check labels on NM when trying to allocate ResourceRequest on the NM with label-expression
> - Respect  labels when calculate headroom/user-limit

This message was sent by Atlassian JIRA

View raw message