hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-291) Dynamic resource configuration on NM
Date Mon, 01 Apr 2013 08:37:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618697#comment-13618697
] 

Luke Lu commented on YARN-291:
------------------------------

bq. The NM is the node controller and responsible for managing the node resources.

There is actually no/zero code in NM itself (other than picking up resource config and report
to RM) so far that cares about how much resource the node has. NM is responsible for managing
the containers that scheduler assigns to it. It does not need to know how much resource it
really has. Only scheduler needs to care about how much resource a node has. Making NM oblivious
to the total resource it has obviate the unnecessary split brain situation.

bq. if it had more work to schedule and there were free resources then it would have already
done so

The point was about scheduling latency. There could be a window of OLTP surge where the nodes
have free resource and large job is submitted. OLTP workload is real time. Also contacting
NM to reconfig resource node by node is inefficient and wasteful.

bq. the currently running tasks will hamper the HBase case even if future work is not scheduled.

This is orthogonal to scheduling, which this JIRA is about. Management agent can suspend/move/kill
the containers if necessary.

bq. What we need is some way to allow HBase higher priority for system resources so that tasks
do not impede HBase perf. Something like YARN-443.

OS scheduling priority works with CPU and to a degree IO (depending on the IO scheduler) but
not memory, which is critical for many OLTP workload.

bq. This JIRA is part of HADOOP-9165 but that one has been resolved as invalid.

I don't know what you are trying to say. HADOOP-9165 is resolved due to wrong JIRA component.
It could be moved to YARN or MAPREDUCE, but the creator chose to file new ones instead.
                
> Dynamic resource configuration on NM
> ------------------------------------
>
>                 Key: YARN-291
>                 URL: https://issues.apache.org/jira/browse/YARN-291
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, scheduler
>            Reporter: Junping Du
>            Assignee: Junping Du
>              Labels: features
>         Attachments: Elastic Resources for YARN-v0.2.pdf, YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch, YARN-291-JMXInterfaceOnNM-02.patch,
YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch, YARN-291-YARNClientCommandline-04.patch
>
>
> The current Hadoop YARN resource management logic assumes per node resource is static
during the lifetime of the NM process. Allowing run-time configuration on per node resource
will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist
with other workloads on the same hardware efficiently, whether or not the environment is virtualized.
About more background and design details, please refer: HADOOP-9165.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message