hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5323) Policies APIs (for Router and AMRMProxy policies)
Date Tue, 23 Aug 2016 19:06:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433409#comment-15433409

Carlo Curino commented on YARN-5323:

[~wangda] thanks for reviewing.

Answering your questions:
 # The AMRMProxyFederationPolicy is used in the AMRMProxyService to determine where to send
the ResourceRequests, while the RouterFederationPolicy is used by the router to determine
where to send the AM. This patch is a bit of a "leaf" patch, other patches will use this code,
so it make sense you don't get much context from this patch alone. 
 # We could remove it, but the key intuition we have is that the two AMRMProxyPolicy and RouterPolicy,
while completely independent (run on different nodes and do different things), need to work
in collaboration with each other to achieve a consistent system behavior. We are thus trying
to use the code structure (e.g., FederationPolicy) as a way to tie them together. This should
help administrators to avoid some of the obvious mistakes (e.g., combining a load-balancing
RouterPolicy with a fault-tolerance focused AMRMProxyPolicy)---if you want you can, but have
to implement a class and configure the system to use it, which should make you think harder
about what you are doing. Also we are using this writer + getter api as a way to handle the
entire lifecycle of ser/deser of the policy configuration through the store. By construction
we want the configuration to be rather opaque (expandibility), but we need to keep the lifecycle
management sane, this is our attempt. 
 # Works for me, though I don't think is the common way for other "*Context" objects around
 # The Router routes more than the APPs, all client-to-rm protocols go through there... Most
of the decisions of where to route are not policy based, as they are deterministic (e.g.,
if you send a kill should go where the app was running). I think the policy name should say
Router to indicate that is used by the router. Also in longer term I can see more policy behaviors
to be added to the router, as we become more sophisticated and operations that are fully manual
today (capacity allocation to subclusters) get automated. 

I hope this help clarify your concerns.

> Policies APIs (for Router and AMRMProxy policies)
> -------------------------------------------------
>                 Key: YARN-5323
>                 URL: https://issues.apache.org/jira/browse/YARN-5323
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>    Affects Versions: YARN-2915
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: YARN-5323-YARN-2915.05.patch, YARN-5323.01.patch, YARN-5323.02.patch,
YARN-5323.03.patch, YARN-5323.04.patch
> This JIRA tracks APIs for the policies that will guide the Router and AMRMProxy decisions
on where to fwd the jobs submission/query requests as well as ResourceRequests.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message