hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
Date Fri, 24 Jul 2015 23:25:06 GMT

     [ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Arun Suresh updated YARN-2154:
    Attachment: YARN-2154.1.patch

Attaching a proof-of-concept patch.

The patch introduces an extra stage in the current preemption logic. Salient points to note
# In the first stage, we iterate through starved LeafQueues and obtain an aggregate  of the
{{ResourceDeficit}}, which has two fields:
## unmarked resources : The deficit for which the queue is starved for, essentially, no app
can be allocated to the queue due to the deficit
## marked resources : These are app specific deficits, viz. node specific resources that an
app is waiting on to launch a container.
# In the second stage, we try to match the Marked resources obtained in the first step with
containers owned by apps that are consuming above their fair/min share. If we find such a
## we first see if any app is already reserved on the Node hosting the container.
## If no, we Reserve the app originating the resource Request on the Node
## we then place the container in the {{warnedContainers}} list
## we return the totalResources that we reclaimed
# In the last stage, we call {{preemptResources}} as before.. with the unmarked resources
+ the reclaimed resources in the previous stage. At which time, the {{warnedContainers}} list
will be iterated over and containers will be killed.

# The Matching can happen more efficiently. In the current patch, all first matching container
that fits a resourcereq is targeted for preemption. This can probably be modified to a best
fit algorithm 
# Fixing test cases.

> FairScheduler: Improve preemption to preempt only those containers that would satisfy
the incoming request
> ----------------------------------------------------------------------------------------------------------
>                 Key: YARN-2154
>                 URL: https://issues.apache.org/jira/browse/YARN-2154
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 2.4.0
>            Reporter: Karthik Kambatla
>            Assignee: Arun Suresh
>            Priority: Critical
>         Attachments: YARN-2154.1.patch
> Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only
preempt resources that would satisfy the incoming request. 

This message was sent by Atlassian JIRA

View raw message