hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers
Date Wed, 13 Jan 2016 21:12:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096987#comment-15096987

Karthik Kambatla commented on YARN-1011:

bq. Does this mean a queue with SLA sensitive jobs could get OPPORTUNISTIC containers?
You bring up a good point. 

Let us consider an example - an app requests for a container at time 0 that would run for
30 seconds. Today, let us say, the app would have gotten the container 10s from submission.
With opportunistic scheduling, let us say we are able to schedule an OPPORTUNISTIC container
at 5s. Now, if we have resource contention within the next 5 seconds, we would just preempt
this OPPORTUNISTIC container and it would get scheduled at 10s as it would have otherwise.
If the contention were to surface at 15s from submission, we would have to preempt and re-allocate
losing 5s compared to what would have happened today. I was hoping the ability to tune allocation
and preemption thresholds would greatly reduce the likelihood of preemptions, but we can't
rely on that for SLA jobs. 

I am wary of allowing applications to ask for only GUARANTEED containers, as that would complicate
the scheduler significantly. Since applications are notified of the ExecutionType, is it okay
to leave it upto them to reject OPPORTUNISTIC containers. We could add this to AMRMClient
and MapReduce.

> [Umbrella] Schedule containers based on utilization of currently allocated containers
> -------------------------------------------------------------------------------------
>                 Key: YARN-1011
>                 URL: https://issues.apache.org/jira/browse/YARN-1011
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Arun C Murthy
>         Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf, yarn-1011-design-v2.pdf
> Currently RM allocates containers and assumes resources allocated are utilized.
> RM can, and should, get to a point where it measures utilization of allocated containers
and, if appropriate, allocate more (speculative?) containers.

This message was sent by Atlassian JIRA

View raw message