hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-291) Dynamic resource configuration
Date Mon, 01 Apr 2013 23:11:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13619298#comment-13619298

Alejandro Abdelnur commented on YARN-291:

bq. How so? In fact ...

Changes in NM capacity triggered from outside of the regular scheduling would unbalance existing
distribution of allocations potentially triggering preemption. You'd need to handle this specially
in the RM/scheduler to handle such scenarios.

bq. First this requires modifying existing applications to support new request API...

Not really, the API would be augmented in a compatible way to support what I'm describing,
existing applications should not be affected as they don't use this new functionality.

bq. Second, RM scheduling is fundamentally high latency ....

It depends how you design you AM that handles unmanaged containers. You could request several
small resources on peak and then release them as you don't need them.

bq. Note, the existing simple patch doesn't change existing API/protocols.

It is adding a new one, that is a change.

> Dynamic resource configuration
> ------------------------------
>                 Key: YARN-291
>                 URL: https://issues.apache.org/jira/browse/YARN-291
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager, scheduler
>            Reporter: Junping Du
>            Assignee: Junping Du
>              Labels: features
>         Attachments: Elastic Resources for YARN-v0.2.pdf, YARN-291-AddClientRMProtocolToSetNodeResource-03.patch,
YARN-291-all-v1.patch, YARN-291-core-HeartBeatAndScheduler-01.patch, YARN-291-JMXInterfaceOnNM-02.patch,
YARN-291-OnlyUpdateWhenResourceChange-01-fix.patch, YARN-291-YARNClientCommandline-04.patch
> The current Hadoop YARN resource management logic assumes per node resource is static
during the lifetime of the NM process. Allowing run-time configuration on per node resource
will give us finer granularity of resource elasticity. This allows Hadoop workloads to coexist
with other workloads on the same hardware efficiently, whether or not the environment is virtualized.
About more background and design details, please refer: HADOOP-9165.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message