hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-291) Dynamic resource configuration on NM
Date Wed, 02 Jan 2013 19:02:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13542297#comment-13542297
] 

Luke Lu commented on YARN-291:
------------------------------

A simpler and more straightforward alternative would be changing the node resource on RM directly
via JMX. No protocol changes would be necessary. Changing resource on NM and propagating the
change to RM would lead to nasty race conditions where RM still thinks that an NM has enough
resource and schedule new containers on the already downsized NM, which should fail and RM
would need to reschedule the containers elsewhere, which is not pretty, especially when large
number of NMs are affected, even if all the corner cases are handled.

bq. we should do RPC and optionally the web-services to the NM directly. And we should add
command line util too.

Since the only near term user of the feature is *external* admin processes that prefer not
having Hadoop jar dependencies, JMX change alone (a much smaller patch that doesn't impact
major production code paths) would suffice. YARN RPC changes can be in a separate JIRA, when
YARN itself needs to change per node resources. I'm not sure how useful a YARN CLI interface
for this is, given it's impractical for people to manually change per node resources (when
number of nodes is greater than 10). OTOH, I'm fine with having everything in, as long as
it doesn't delay the inclusion of the basic functionality.



                
> Dynamic resource configuration on NM
> ------------------------------------
>
>                 Key: YARN-291
>                 URL: https://issues.apache.org/jira/browse/YARN-291
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, scheduler
>            Reporter: Junping Du
>            Assignee: Junping Du
>              Labels: features
>         Attachments: Elastic Resources for YARN-v0.2.pdf, YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch, YARN-291-JMXInterfaceOnNM-02.patch,
YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch, YARN-291-YARNClientCommandline-04.patch
>
>
> The current Hadoop YARN resource management logic assumes per node resource is static
during the lifetime of the NM process. Allowing run-time configuration on per node resource
will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist
with other workloads on the same hardware efficiently, whether or not the environment is virtualized.
About more background and design details, please refer: HADOOP-9165.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message