hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vrushali C (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-3167) [Aggregator implementation] Implement the core functionality of the base aggregator service
Date Wed, 25 Feb 2015 00:29:05 GMT

     [ https://issues.apache.org/jira/browse/YARN-3167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Vrushali C updated YARN-3167:
    Attachment: RM-AM-NM-Aggregator.png

Attaching a couple of diagrams that illustrate the design discussions between me and Sangjin
and Karthik and Robert.

Some notes:
- Per node aggregator process runs as an auxillary service inside the NM
- It has a per application datastructure to receive and write entity info for that app
- The writer API has the following (at the least): 
---- write (entity) : this will write all the contents of the entity to the backend store
---- addEvent (application id, event) : this writes the lifecycle events
---- updateMetrics (application id, timeline metric) : this writes out the metrics
- the base aggregator service api also has the above in its API
- in addition to these, the base aggregator service could do a sync/async write call for each
of the above. 

For the PoC, we could choose to have every write as a synchronous write.

> [Aggregator implementation] Implement the core functionality of the base aggregator service
> -------------------------------------------------------------------------------------------
>                 Key: YARN-3167
>                 URL: https://issues.apache.org/jira/browse/YARN-3167
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>         Attachments: RM-AM-NM-Aggregator.png, Sequence_diagram_User_RM_AM_NM_Aggregator_Writer.png
> The basic skeleton of the timeline aggregator has been set up by YARN-3030. We need to
implement the core functionality of the base aggregator service. The key things include
> - handling the requests from clients (sync or async)
> - buffering data
> - handling the aggregation logic
> - invoking the storage API

This message was sent by Atlassian JIRA

View raw message