hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-291) [Umbrella] Dynamic resource configuration
Date Tue, 03 Dec 2013 02:06:38 GMT

    [ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13837238#comment-13837238

Junping Du commented on YARN-291:

> Junping, just saw your comments on YARN-999 . I can help on it.
Thanks! I plan to finish option without timeout in Dec, so it would be great for you to help
on timeout part.
>By different options above, do you mean overCommitTimeoutMills > 0, = 0, <0 ? I
want to find out more use cases associated with this setting besides >graceful decommission.
For example, you mentioned preemption for long running tasks in YARN-999, is that part of
or a different use case from graceful >decommission?
Yes. overCommitTimeoutMills value sets different options here. <0 (or just -1) means we
tolerant tasks running to the end even under resource over-consumed cases; >=0 means we
only tolerant a few time specified in overCommitTimeoutMills. Once timeout, we do aggressive
ways (i.e. preemption on assigned containers with frozen or kill tasks) to reclaim resources
so that NM's resource can get it balanced again. Graceful decommission is just a special case
for this where we always set NM's totalResource to 0 first, so all assigned containers will
get released after a timeout (except timeout = -1). If we can set a proper timeout value here,
then it will get chance for NM to finish running tasks with intermediate map output get retrieval
before decommissioned and that's why we call it "graceful".  
>Also, about the August patch CoreAndAdmin.patch (in YARN-291) , can you let us know your
plan about it because it seems useful for graceful 
> decommission from outside of YARN code.
Most of patches are on the track. YARN-311 (core changes) get checked in, YARN-312 (RPC) get
reviewed with +1. Will be there soon.

> [Umbrella] Dynamic resource configuration
> -----------------------------------------
>                 Key: YARN-291
>                 URL: https://issues.apache.org/jira/browse/YARN-291
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, scheduler
>            Reporter: Junping Du
>            Assignee: Junping Du
>              Labels: features
>         Attachments: Elastic Resources for YARN-v0.2.pdf, YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
YARN-291-CoreAndAdmin.patch, YARN-291-JMXInterfaceOnNM-02.patch, YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch,
YARN-291-YARNClientCommandline-04.patch, YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch
> The current Hadoop YARN resource management logic assumes per node resource is static
during the lifetime of the NM process. Allowing run-time configuration on per node resource
will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist
with other workloads on the same hardware efficiently, whether or not the environment is virtualized.
More background and design details can be found in attached proposal.

This message was sent by Atlassian JIRA

View raw message