hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dian Fu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
Date Mon, 28 Sep 2015 17:11:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14933584#comment-14933584

Dian Fu commented on YARN-3964:

Hi [~Naganarasimha] and [~sunilg],
Thanks for your comments. 
 I feel as nodes are more in cluster, and response time from {{nodeLabelsMappingProvider.getNodeLabels}}
may also be more.
{{updateNodeLabelsInternal}} is called in two places: one place is in thread {{RMDelegatedNodeLabelsUpdaterTimerTask}}
and the other place is on register. 
* When called in thread {{RMDelegatedNodeLabelsUpdaterTimerTask}}, the impact is minor as
it is working in a separate thread and so the response time of {{RMDelegatedNodeLabelsUpdaterTimerTask}}
won't impact other threads. Further more, only one call of {{nodeLabelsMappingProvider.getNodeLabels}}
can fetch all the node labels and so the response time should not be too long.
* When called on register, the response time of {{nodeLabelsMappingProvider.getNodeLabels}}
does have impact as it may lead to longer register time. That's the reason why patch 006 doesn't
update node labels synchronously on register. Thoughts? 
This piece of code i think should be done by the caller handle(SchedulerEvent event) rather
than updateLabelsOnNode as its synchronized method.{quote}
Make sense. 

> Support NodeLabelsProvider at Resource Manager side
> ---------------------------------------------------
>                 Key: YARN-3964
>                 URL: https://issues.apache.org/jira/browse/YARN-3964
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Dian Fu
>            Assignee: Dian Fu
>         Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, YARN-3964.003.patch,
YARN-3964.004.patch, YARN-3964.005.patch, YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch,
YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch, YARN-3964.011.patch, YARN-3964.012.patch,
> Currently, CLI/REST API is provided in Resource Manager to allow users to specify labels
for nodes. For labels which may change over time, users will have to start a cron job to update
the labels. This has the following limitations:
> - The cron job needs to be run in the YARN admin user.
> - This makes it a little complicate to maintain as users will have to make sure this
service/daemon is alive.
> Adding a Node Labels Provider in Resource Manager will provide user more flexibility.

This message was sent by Atlassian JIRA

View raw message