hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subru Krishnan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6511) Federation Intercepting and propagating AM-RM communications (part two: secondary subclusters added)
Date Fri, 02 Jun 2017 20:32:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035370#comment-16035370

Subru Krishnan commented on YARN-6511:

Thanks [~botong] for the patch. I looked at it & please find my comments below:
  * I don't see the need for {{TestableAMRMProxyPolicy}}, we should simply reuse {{BroadcastAMRMProxyPolicy}}.
If possible, do the same for {{TestableRouterPolicy}}.
  * Do we need a {{UnmanagedAMPoolManager}} per interceptor instance or can we use one at
{{AMRMProxyService}} level?
  * *loadAMRMPolicy* is independent of {{FederationInterceptor}}, can be refactored out to
common class.
  * Is updating the _queue_ below safe in *loadAMRMPolicy*?
      {code} queue = YarnConfiguration.DEFAULT_FEDERATION_POLICY_KEY;{code}
  * I feel the *finishApplicationMaster* of the pool should be moved to {{UnmanagedAMPoolManager}}.
  * I see dynamic instantiations of {{ExecutorCompletionService}} in finish, register, etc
invocations. Wouldn't we be better served by pre-initializing it?
  * Is *getSubClusterForNode* required as the resolver should be doing this instead of every
  * Consider looping on _registrations_ in lieu of _requests_ in *sendRequestsToSecondaryResourceManagers*
 as that'll not only minimize the iterations but also make the checks redundant.
  * Quite a few ops like creation of _YarnConfiguration_, _getApplicationContext_ invocation
etc can be moved outside the for loop in *registerWithNewSubClusters*.
  * {{YarnConfiguration.getClusterId(getConf())}} is unnecessary as we have _homeSubClusterId_.

> Federation Intercepting and propagating AM-RM communications (part two: secondary subclusters
> ----------------------------------------------------------------------------------------------------
>                 Key: YARN-6511
>                 URL: https://issues.apache.org/jira/browse/YARN-6511
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Botong Huang
>            Assignee: Botong Huang
>         Attachments: YARN-6511-YARN-2915.v1.patch, YARN-6511-YARN-2915.v2.patch
> In order to support transparent "spanning" of jobs across sub-clusters, all AM-RM communications
are proxied (via YARN-2884).
> This JIRA tracks federation-specific mechanisms that decide how to "split/broadcast"
requests to the RMs and "merge" answers to 
> the AM.
> This the part two jira, which adds secondary subclusters and do full split-merge for
requests. Part one is in YARN-3666

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message