hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantinos Karanasos (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2884) Proxying all AM-RM communications
Date Fri, 21 Nov 2014 15:20:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221025#comment-14221025

Konstantinos Karanasos commented on YARN-2884:

[~kasha], [~curino], [~subru], given that this proxy/agent will only focus on the AM-RM communication,
we may also explicitly call it AMRMProxy or AMRMAgent (following the naming convention of
the already existing AMRMClient* classes).

[~djp] I just added a comment in the umbrella JIRA (YARN-2877), trying to give some more details.
We are not proposing to substitute all scheduling decisions with distributed ones. The guaranteed-start
containers will continue to be scheduled by the central RM. However, the queueable ones will
be scheduled in a distributed fashion. 
The first candidate for queueable containers is the short-running tasks, in which the overhead
of contacting the central RM is a significant part of the overall task execution time. Scheduling
these requests without contacting the central RM will reduce their latency, increase the utilization
of the cluster (no idle resources waiting to contact the RM), while it will offload the central
RM (which is good for scaling in big clusters).

> Proxying all AM-RM communications
> ---------------------------------
>                 Key: YARN-2884
>                 URL: https://issues.apache.org/jira/browse/YARN-2884
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Carlo Curino
> We introduce the notion of an RMProxy, running on each node (or once per rack). Upon
start the AM is forced (via tokens and configuration) to direct all its requests to a new
services running on the NM that provide a proxy to the central RM. 
> This give us a place to:
> 1) perform distributed scheduling decisions
> 2) throttling mis-behaving AMs
> 3) mask the access to a federation of RMs

This message was sent by Atlassian JIRA

View raw message