hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1039) Add parameter for YARN resource requests to indicate "long lived"
Date Wed, 28 Jan 2015 02:34:36 GMT

    [ https://issues.apache.org/jira/browse/YARN-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294613#comment-14294613

Chris Douglas commented on YARN-1039:

[~cwelch] YARN shouldn't understand the lifecycle for a service or the progress/dependencies
for task containers. As proposed, an AM will receive a lease on a container for some duration.
Before the lease expires, it can relinquish the lease or request that it be renewed. While
this adds some complexity in the AM implementation- it needs to track and renew its container
leases- it's mostly library code that admits straightforward, naive implementations. The most
obvious strawman would request all resources at the longest possible duration and always renew.

Mapping an enumeration expressing an AM lifecycle into a policy for requesting, refreshing,
and managing resources is an excellent client-side abstraction. Even if an implementation
of YARN only receives (and only issues) leases from a fixed set of values, the underlying
abstraction can admit arbitrary durations. An enumeration is a good API for applications,
but it's the RM framework could have a more fine-grained substrate.

Leases actually help services run under YARN. By way of example, refusing to renew a lease
could signal that the node will be decommissioned, or that some cluster-wide invariant- like
balanced utilization or fairness- is better met by (re)moving that container. Refusing to
renew a lease- or renewing it for a shorter period- could signal the service to request new

> Add parameter for YARN resource requests to indicate "long lived"
> -----------------------------------------------------------------
>                 Key: YARN-1039
>                 URL: https://issues.apache.org/jira/browse/YARN-1039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0, 2.1.1-beta
>            Reporter: Steve Loughran
>            Assignee: Craig Welch
>         Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
> A container request could support a new parameter "long-lived". This could be used by
a scheduler that would know not to host the service on a transient (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived containers
on the same node

This message was sent by Atlassian JIRA

View raw message