hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2498) [YARN-796] Respect labels in preemption policy of capacity scheduler
Date Sun, 21 Sep 2014 17:01:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142513#comment-14142513

Sunil G commented on YARN-2498:

Hi [~wangda]

Great work Wangda. Very good design to solve the issues. I have few doubts though :)

bq. 2G resource in node1 with label x, and queueA has label y, so queueA cannot access node1
A scenario where *node1* has more than 50% (say 60) of cluster resources, and queue A is given
50% in CS. IN that case, is there any chance of under utilization?

2. When two or more nodes are merged to a single resource tree, and if these nodes share some
common labels, i could see a common total sum is been finally stored per label in the consolidated

Here I feel, we may need to split up the resource of label in each node level. 

For eg; label *x* and *y* are needed for *queueA*. label *y* and *z* are needed for *queueB*.
And if the label *y* is shared between 2 or more nodes, I am not sure whether it will cause
some pblm.

bq.(container.resource can be used by pendingResourceLeaf)

For preemption, we just calculate to match the totalResourceToPreempt from the over utilized
queues. But whether this container is from which node, and also under which label, and whether
this label is coming under which queue. Do we need to do this check for each container?

> [YARN-796] Respect labels in preemption policy of capacity scheduler
> --------------------------------------------------------------------
>                 Key: YARN-2498
>                 URL: https://issues.apache.org/jira/browse/YARN-2498
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-2498.patch, YARN-2498.patch, YARN-2498.patch, yarn-2498-implementation-notes.pdf
> There're 3 stages in ProportionalCapacityPreemptionPolicy,
> # Recursively calculate {{ideal_assigned}} for queue. This is depends on available resource,
resource used/pending in each queue and guaranteed capacity of each queue.
> # Mark to-be preempted containers: For each over-satisfied queue, it will mark some containers
will be preempted.
> # Notify scheduler about to-be preempted container.
> We need respect labels in the cluster for both #1 and #2:
> For #1, when there're some resource available in the cluster, we shouldn't assign it
to a queue (by increasing {{ideal_assigned}}) if the queue cannot access such labels
> For #2, when we make decision about whether we need preempt a container, we need make
sure, resource this container is *possibly* usable by a queue which is under-satisfied and
has pending resource.

This message was sent by Atlassian JIRA

View raw message