hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1434) Single Job can affect fairshare of others
Date Thu, 21 Nov 2013 21:37:35 GMT

    [ https://issues.apache.org/jira/browse/YARN-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13829377#comment-13829377

Carlo Curino commented on YARN-1434:

This has been observed while modifying the mapreduce AM behavior for other reasons. If the
AM aggressively returns containers, it seems to be able to create the illusion to be under-capacity
while wasting resources for everyone. A second job running in a separate queue (which was
supposed to receive 50% of the cluster resources) was starved (only getting about 30% of the
resources). This should be confirmed independently as the environment we observed this in
had too much going on (i.e., this might be a false positive). 

If confirmed, this might be quite bad, as a single malevolent AM could affect the cluster
utilization possibly by a lot.
[~sandyr], [~acmurthy]  thoughts?

> Single Job can affect fairshare of others
> -----------------------------------------
>                 Key: YARN-1434
>                 URL: https://issues.apache.org/jira/browse/YARN-1434
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>            Reporter: Carlo Curino
>            Priority: Minor
> A job receiving containers and deciding not to use them and yielding them back in the
next heartbeat could significantly affect the amount of resources given to other jobs. 
> This is because by yielding containers back the job appears always to be under-capacity
(more than others) so it is picked to be the next to receive containers.
> Observed by Robert Grandl, to be independently confirmed.

This message was sent by Atlassian JIRA

View raw message