hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cindy Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-291) [Umbrella] Dynamic resource configuration
Date Thu, 14 Nov 2013 01:10:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13822052#comment-13822052

Cindy Li commented on YARN-291:


I'm trying to find out where you handle different cases of commitTimeout, and getting to this
in schedulerUtils.java:

      // TODO process resource over-commitment case (allocated containers
      // > total capacity) in different option by getting value of
      // overCommitTimeoutMillis.

What's your thought on this? Are you working on it? 

In our case, we would need to set overCommitTimeoutMills to be >0,  and the capacity(memory/vore)
to be zero for node to be put into maintenance. What I would expect is:
#1. The scheduler stops scheduling new application attempts to the node
#2. Clean up containers on that node after overCommitTimeoutMills. 

>From my understanding, #1 is done once the capacity is set to 0, but what do you plan
to put there for #2? YARN-999 seems not solving #2.

> [Umbrella] Dynamic resource configuration
> -----------------------------------------
>                 Key: YARN-291
>                 URL: https://issues.apache.org/jira/browse/YARN-291
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, scheduler
>            Reporter: Junping Du
>            Assignee: Junping Du
>              Labels: features
>         Attachments: Elastic Resources for YARN-v0.2.pdf, YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
YARN-291-CoreAndAdmin.patch, YARN-291-JMXInterfaceOnNM-02.patch, YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch,
YARN-291-YARNClientCommandline-04.patch, YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch
> The current Hadoop YARN resource management logic assumes per node resource is static
during the lifetime of the NM process. Allowing run-time configuration on per node resource
will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist
with other workloads on the same hardware efficiently, whether or not the environment is virtualized.
More background and design details can be found in attached proposal.

This message was sent by Atlassian JIRA

View raw message