hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3942) Timeline store to read events from HDFS
Date Fri, 28 Aug 2015 22:05:47 GMT

    [ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14720668#comment-14720668

Jason Lowe commented on YARN-3942:

Yeah that's going to be tricky, especially if we need to move most of the code into YARN.
 Haven't had time to give this much thought, but the only way I can think of to keep most
of the functionality in YARN is to have the timeline client be able to specify when a new
"session" starts (i.e.: entity file writer should start writing to a new file and user provides
some clue/hint as to what to name the file).  We can then have a plugin on the entity file
server side that allows apps to override the getTimelineStoreForRead functionality.

If that was in place then the Tez side could start a new session (dag file) each time the
dag changed.  The Tez-specific plugin on the timeline server side could then translate dag/vertex/task/attempt
IDs into the appropriate dag file to cache.  There would still be some questions as to how
the timeline store cache would be managed on the server side and how to support multiple framework-specific
plugins simultaneously.

> Timeline store to read events from HDFS
> ---------------------------------------
>                 Key: YARN-3942
>                 URL: https://issues.apache.org/jira/browse/YARN-3942
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: timelineserver
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-3942.001.patch
> This adds a new timeline store plugin that is intended as a stop-gap measure to mitigate
some of the issues we've seen with ATS v1 while waiting for ATS v2.  The intent of this plugin
is to provide a workable solution for running the Tez UI against the timeline server on a
large-scale clusters running many thousands of jobs per day.

This message was sent by Atlassian JIRA

View raw message