hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-291) Dynamic resource configuration on NM
Date Mon, 01 Apr 2013 19:09:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619059#comment-13619059
] 

Alejandro Abdelnur commented on YARN-291:
-----------------------------------------

+1 on the idea of dynamic resource configuration. The sharing w/HBase use case is a scenario
we see quite often.

Though, I have concerns on changing the available NMs capacity directly in the NMs. This would
complicate significantly allocation changes/re-computation of resources for running applications.


NM available resources discovery should be static as it is completely acceptable to restart
them without compromising running apps:

* A NM on startup detects the OS for its CPU/memory and reports this to the RM
** config properties in the NM, if present, could override OS detection values
* A NM should be restarted to report a new configuration


To achieve dynamic resource configuration we should fully leverage YARN resource scheduling
framework. For example, this would be done by introducing the concept of 'unmanaged resources'.
The idea would be something like this:

* An (unmanaged) AM requests unmanaged resources from the RM (a new flag in a resource request)
* The RM assigns the unmanaged resource to a NM
* The RM notifies the NM of the unmanaged resource assignment
* The RM notifies the AM of the unmanaged resource assignment
* The unmanaged assigned resource does not time out in the RM/scheduler due to container not
starting (Bikas, our chat in the last yarn meet-up clicked)
* The AM will, out of band, claim those resources from the corresponding node
* The resources will remain assigned to the AM until the AM releases them, the AM goes MIA
or the resources are preempted

This together with YARN-392 (enforce locality for a request), will allow an external component
to claim cluster resources while enforcing existing YARN resource allocation policies and
priorities.

With this approach the changes in the RM/scheduler and API are quite simple.

Thoughts?

                
> Dynamic resource configuration on NM
> ------------------------------------
>
>                 Key: YARN-291
>                 URL: https://issues.apache.org/jira/browse/YARN-291
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, scheduler
>            Reporter: Junping Du
>            Assignee: Junping Du
>              Labels: features
>         Attachments: Elastic Resources for YARN-v0.2.pdf, YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch, YARN-291-JMXInterfaceOnNM-02.patch,
YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch, YARN-291-YARNClientCommandline-04.patch
>
>
> The current Hadoop YARN resource management logic assumes per node resource is static
during the lifetime of the NM process. Allowing run-time configuration on per node resource
will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist
with other workloads on the same hardware efficiently, whether or not the environment is virtualized.
About more background and design details, please refer: HADOOP-9165.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message