hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3044) [Event producers] Implement RM writing app lifecycle events to ATS
Date Wed, 25 Mar 2015 19:00:54 GMT

    [ https://issues.apache.org/jira/browse/YARN-3044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14380518#comment-14380518
] 

Vinod Kumar Vavilapalli commented on YARN-3044:
-----------------------------------------------

It's not just events. We need to model entities, their information, events, metrics and configuration.
The obvious list of entities is Application, ApplicationAttempt and Container.

Application and ApplicationAttempt are easy to tackle and scalable for all their data.

Container information has scalability concerns what with thousands of containers getting allocated
and finishing every second in a big enough cluster. Again, just the container life-cycle events
are okay, but sending metrics for each container (a very important use-case) is a challenge.
Assuming just the metric snapshots, we are talking about 5K containers * 10 metrics = 50K
writes per second on a big cluster. If we also have to track time-series (I want to), it's
( 50K writes per second ) * (Container duration) number of events on storage.


> [Event producers] Implement RM writing app lifecycle events to ATS
> ------------------------------------------------------------------
>
>                 Key: YARN-3044
>                 URL: https://issues.apache.org/jira/browse/YARN-3044
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Naganarasimha G R
>
> Per design in YARN-2928, implement RM writing app lifecycle events to ATS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message