hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1957) ProportionalCapacitPreemptionPolicy handling of corner cases...
Date Thu, 24 Apr 2014 05:17:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979295#comment-13979295

Carlo Curino commented on YARN-1957:

Attached a patch fixing the issues above. Conceptually we compute the fix-point allocation
for non-zero queues, and then spread the remaining capacity uniformly among zero-size queues.
Also each "offer" to a queue now handles maxCapacity.

The old behavior for zero-size queues (i.e., preemption of all containers regardless to external
pressure) should be possible to obtain by setting maxCapacity = capacity = 0; 

Test have been updated to verify maxCapacity behavior as well. We also tested on a medium
size cluster running gridmix with lots of preemptions and it seems to behave as expected,

however a careful code review from someone else is more than welcome. 

> ProportionalCapacitPreemptionPolicy handling of corner cases...
> ---------------------------------------------------------------
>                 Key: YARN-1957
>                 URL: https://issues.apache.org/jira/browse/YARN-1957
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-1957.patch, YARN-1957_test.patch
> The current version of ProportionalCapacityPreemptionPolicy should be improved to deal
with the following two scenarios:
> 1) when rebalancing over-capacity allocations, it potentially preempts without considering
the maxCapacity constraints of a queue (i.e., preempting possibly more than strictly necessary)
> 2) a zero capacity queue is preempted even if there is no demand (coherent with old use
of zero-capacity to disabled queues)
> The proposed patch fixes both issues, and introduce few new test cases.

This message was sent by Atlassian JIRA

View raw message