hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subru Krishnan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2884) Proxying all AM-RM communications
Date Fri, 24 Jul 2015 01:26:06 GMT

    [ https://issues.apache.org/jira/browse/YARN-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639788#comment-14639788
] 

Subru Krishnan commented on YARN-2884:
--------------------------------------

Thanks [~kishorch] for posting the patch. I have a few comments:
   * The current implementation assumes that we will set the AMRMProxy address as the RM scheduler
address in the client configuration. This will work for _MapReduce_, _Spark_ etc where the
client configuration is passed to the AM. But we need to explicitly override the RM scheduler
address via the container launch environment to allow proxying more generically to work with
all AMs like _DistributedShell_, _REEF_, etc  
   * In _AMRMProxyService_, *authorizeRequest* is the exact same check as done by _ApplicationMasterService_
so it'll be better to refactor the code to reuse for manageability.
   * Can we use the *AMRMTokenSelector* to select the AMRMToken in _AMRMProxyService_.
   * I see that the *MasterKeyRoller* is used in multiple places. We should have a _RolloverSecretManager_
that does the rollover and have _AMRMProxyTokenSecretManager_ (and others) extend it.

There are few test patch issues in the first version of the patch, looks mostly to do with
whitespaces. Can you take a look & fix those in the next iteration.

> Proxying all AM-RM communications
> ---------------------------------
>
>                 Key: YARN-2884
>                 URL: https://issues.apache.org/jira/browse/YARN-2884
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Carlo Curino
>            Assignee: Kishore Chaliparambil
>         Attachments: YARN-2884-V1.patch
>
>
> We introduce the notion of an RMProxy, running on each node (or once per rack). Upon
start the AM is forced (via tokens and configuration) to direct all its requests to a new
services running on the NM that provide a proxy to the central RM. 
> This give us a place to:
> 1) perform distributed scheduling decisions
> 2) throttling mis-behaving AMs
> 3) mask the access to a federation of RMs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message