hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2073) FairScheduler starts preempting resources even with free resources on the cluster
Date Wed, 21 May 2014 03:34:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004290#comment-14004290

Sandy Ryza commented on YARN-2073:

There are some situations where preemption with free resources on the cluster is the right
thing to do.

For example, if I'm requesting 2 GB containers, I have no resources, and 100 nodes on the
cluster each have 1GB remaining, containers should get preempted on my behalf.

There are also cases arising from requests with strict locality - the cluster might have resources
available because I'm waiting on a subset of nodes.  (In this case, we'd probably want to
make sure preemption only happens on the nodes being waited for; otherwise we'd kill containers

If the goal is to make sure that we aren't preempting on behalf of an application that's actually
receiving resources, it might also be worth considering time-based approaches. E.g. only preempt
on behalf of an application that hasn't received resources in some amount of time.

> FairScheduler starts preempting resources even with free resources on the cluster
> ---------------------------------------------------------------------------------
>                 Key: YARN-2073
>                 URL: https://issues.apache.org/jira/browse/YARN-2073
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 2.4.0
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>            Priority: Critical
>         Attachments: yarn-2073-0.patch, yarn-2073-1.patch, yarn-2073-2.patch
> Preemption should kick in only when the currently available slots don't match the request.

This message was sent by Atlassian JIRA

View raw message