hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Roberts (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1011) [Umbrella] Schedule containers based on utilization of currently allocated containers
Date Wed, 13 Jan 2016 20:33:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096938#comment-15096938

Nathan Roberts commented on YARN-1011:

Thanks for the update [~kasha]. I have a few questions but I'll start at a high level since
the answers may clear up others.
bq. On SchedulerNode#allocateContainer, mark the container OPPORTUNISTIC if allocating this
container would take the allocation over the advertised capacity.  And, keep track of opportunistic
containers in a list separate from launchedContainers.

It sounds like any container from any queue (assuming CapacityScheduler here but similar constructs
will exist in other schedulers) could be marked OPPORTUNISTIC. Does this mean a queue with
SLA sensitive jobs could get OPPORTUNISTIC containers? I just don't see how these jobs will
be able to meet their SLAs with the uncertainty as to the resources they'll actually get,
and the likelihood they'll get shot by the NM. This is precisely the reason there is the ability
to designate queues as preemptable since SLA sensitive jobs just don't do well in preemtable

> [Umbrella] Schedule containers based on utilization of currently allocated containers
> -------------------------------------------------------------------------------------
>                 Key: YARN-1011
>                 URL: https://issues.apache.org/jira/browse/YARN-1011
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Arun C Murthy
>         Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf, yarn-1011-design-v2.pdf
> Currently RM allocates containers and assumes resources allocated are utilized.
> RM can, and should, get to a point where it measures utilization of allocated containers
and, if appropriate, allocate more (speculative?) containers.

This message was sent by Atlassian JIRA

View raw message