hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ram Venkatesh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2362) Capacity Scheduler apps with requests that exceed capacity can starve pending apps
Date Sun, 27 Jul 2014 05:21:38 GMT

    [ https://issues.apache.org/jira/browse/YARN-2362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14075554#comment-14075554
] 

Ram Venkatesh commented on YARN-2362:
-------------------------------------

I agree that apps that need the entire cluster capacity  are likely not common. However, I
think the scenario above can happen in busy clusters where an app might make a request that
exceeds _current_ capacity and hence block all other apps. Yes, whenever more resources get
freed up and App1's request is satisfied, only then will App2 run. Note, since we are enumerating
the set of active apps, the behavior is actually non-deterministic - if the new app happens
to be enumerated before the large app, the allocation request will actually be satisfied.
The change proposed here makes it deterministic and can also reduce the wait for jobs that
can complete - the downside of course is the large app can now experience starvation if small
apps keep getting through. 

> Capacity Scheduler apps with requests that exceed capacity can starve pending apps
> ----------------------------------------------------------------------------------
>
>                 Key: YARN-2362
>                 URL: https://issues.apache.org/jira/browse/YARN-2362
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>    Affects Versions: 2.4.1
>            Reporter: Ram Venkatesh
>
> Cluster configuration:
> Total memory: 8GB
> yarn.scheduler.minimum-allocation-mb 256
> yarn.scheduler.capacity.maximum-am-resource-percent 1 (100%, test only config)
> App 1 makes a request for 4.6 GB, succeeds, app transitions to RUNNING state. It subsequently
makes a request for 4.6 GB, which cannot be granted and it waits.
> App 2 makes a request for 1 GB - never receives it, so the app stays in the ACCEPTED
state for ever.
> I think this can happen in leaf queues that are near capacity.
> The fix is likely in LeafQueue.java assignContainers near line 861, where it returns
if the assignment would exceed queue capacity, instead of checking if requests for other active
applications can be met.
> {code:title=LeafQueue.java|borderStyle=solid}
>            // Check queue max-capacity limit
>            if (!assignToQueue(clusterResource, required)) {
> -            return NULL_ASSIGNMENT;
> +            break;
>            }
> {code}
> With this change, the scenario above allows App 2 to start and finish while App 1 continues
to wait.
> I have a patch available, but wondering if the current behavior is by design.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message