hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3039) [Aggregator wireup] Implement ATS app-appgregator service discovery
Date Thu, 12 Mar 2015 21:20:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359441#comment-14359441

Zhijie Shen commented on YARN-3039:

bq. When AM get launched, NM auxiliary service will add a new aggregator service to aggregatorCollection
(per Node) for necessary binding work. aggregatorCollection also has a client for AggregatorNodeManagerProtocol
to notify NM on new app aggregator registered and detailed address.

Hi Junping, thanks for creating the new patch. Sorry for raising the question in late, but
I'd like to think it out loudly about the first step. Nowadays, app-level aggregator is started
by the callback handler listening to the container start even of NM. Given we are going to
support stand-alone and container mode, this approach may not work. As we're going to have
IPC channel between aggregator and NM, should we use an IPC call to invoke adding one app-level

So the protocol is that NM sends a request to the aggregator collection to start a app-level
aggregator, and collection responds with the aggregator address. However, in this case, it
may not be AggregatorNodemanagerProtocol, but NodemanagerAggregatorProtocol instead. The benefit
is to uniform the way of starting app-level aggregator inside node-level aggregator (at lease
it seem that we need to something similar in YARN-3033),  and further reducing the dependency/assumption
on aux service.

[~vinodkv] and [~sjlee0] how do you think about it?

> [Aggregator wireup] Implement ATS app-appgregator service discovery
> -------------------------------------------------------------------
>                 Key: YARN-3039
>                 URL: https://issues.apache.org/jira/browse/YARN-3039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Junping Du
>         Attachments: Service Binding for applicationaggregator of ATS (draft).pdf, Service
Discovery For Application Aggregator of ATS (v2).pdf, YARN-3039-no-test.patch, YARN-3039-v2-incomplete.patch,
YARN-3039-v3-core-changes-only.patch, YARN-3039-v4.patch, YARN-3039-v5.patch
> Per design in YARN-2928, implement ATS writer service discovery. This is essential for
off-node clients to send writes to the right ATS writer. This should also handle the case
of AM failures.

This message was sent by Atlassian JIRA

View raw message