hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2933) Capacity Scheduler preemption policy should only consider capacity without labels temporarily
Date Thu, 08 Jan 2015 18:41:35 GMT

    [ https://issues.apache.org/jira/browse/YARN-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14269836#comment-14269836
] 

Wangda Tan commented on YARN-2933:
----------------------------------

[~mayank_bansal],
Thanks for updating the patch,

bq. I don't think thats needed as the basic functionality for the test is to demonstrate we
can skip labeled container, So I think it does not mater.
I think it matters, and this test is not only verify we can skip labeled containers, but also
demonstrate we can make ideal_allocation correct. I still suggest to add the one-line change
as I mentioned in https://issues.apache.org/jira/browse/YARN-2933?focusedCommentId=14268631&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14268631
 to verify ideal_allocation is correctly computed.

And for [~sunilg]'s comment, I think Mayank has already answered you.

Thanks,
Wangda



> Capacity Scheduler preemption policy should only consider capacity without labels temporarily
> ---------------------------------------------------------------------------------------------
>
>                 Key: YARN-2933
>                 URL: https://issues.apache.org/jira/browse/YARN-2933
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Wangda Tan
>            Assignee: Mayank Bansal
>         Attachments: YARN-2933-1.patch, YARN-2933-2.patch, YARN-2933-3.patch, YARN-2933-4.patch,
YARN-2933-5.patch, YARN-2933-6.patch
>
>
> Currently, we have capacity enforcement on each queue for each label in CapacityScheduler,
but we don't have preemption policy to support that. YARN-2498 is targeting to support preemption
respect node labels, but we have some gaps in code base, like queues/FiCaScheduler should
be able to get usedResource/pendingResource, etc. by label. These items potentially need to
refactor CS which we need spend some time carefully think about.
> For now, what immediately we can do is allow calculate ideal_allocation and preempt containers
only for resources on nodes without labels, to avoid regression like: A cluster has some nodes
with labels and some not, assume queueA isn't satisfied for resource without label, but for
now, preemption policy may preempt resource from nodes with labels for queueA, that is not
correct.
> Again, it is just a short-term enhancement, YARN-2498 will consider preemption respecting
node-labels for Capacity Scheduler which is our final target. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message