hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3039) [Aggregator wireup] Implement ATS app-appgregator service discovery
Date Wed, 04 Mar 2015 15:58:05 GMT

    [ https://issues.apache.org/jira/browse/YARN-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347053#comment-14347053

Naganarasimha G R commented on YARN-3039:

Hi [~djp],
 bq. for security reason it make NM to take AMRMTokens for using TimelineClient in future
which make less sense. To get rid of rack condition you mentioned above, we propose to use
observer pattern to make TimelineClient can listen aggregator address update in AM or NM (wrap
with retry logic to tolerant connection failure).
Even if we are not able to have "AMRMClient can be wrapped into TimelineClient" i feel other
suggestion from vinod was right 
{{to add a blocking call in AMRMClient to get aggregator address directly from RM.}} instead
of observer pattern @ the AM side.  thoughts ?

bq. There are other ways (check from diagram in YARN-3033) that app aggregators could be deployed
in a separate process or an independent container which make less sense to have a protocol
between AUX service and RM. I think now we should plan to add a protocol between aggregator
and NM, and then notify RM through NM-RM heartbeat on registering/rebind for aggregator.
Yes i have gone through 3033, but earlier was trying to mention as our current approach was
with NM AUX service. But anyway what i wanted was some kind of protocol between appAggregators
with either NM or RM. Protocol between NM and appAgregator should suffice all other ways to
launch AppAgregators.

bq. app aggregator should have logic to consolidate all messages (events and metrics) for
one application into more complex and flexible new data model. If each NM do aggregation separately,
then it still a writer (like old timeline service), but not an aggregator
Well if there is no logic/requirement to aggregate/consolidate all messages (events and metrics)
for an App, then in my opinion it better not to have additional instances of aggregators and
we can  keep it similar to old Timline service.

bq. Will update proposal to reflect all these discussions (JIRA's and offline).
Thanks it will be more clear to implement if we have the proposals documented.

> [Aggregator wireup] Implement ATS app-appgregator service discovery
> -------------------------------------------------------------------
>                 Key: YARN-3039
>                 URL: https://issues.apache.org/jira/browse/YARN-3039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Naganarasimha G R
>         Attachments: Service Binding for applicationaggregator of ATS (draft).pdf, YARN-3039-no-test.patch
> Per design in YARN-2928, implement ATS writer service discovery. This is essential for
off-node clients to send writes to the right ATS writer. This should also handle the case
of AM failures.

This message was sent by Atlassian JIRA

View raw message