hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vrushali C (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3981) support timeline clients not associated with an application
Date Thu, 29 Sep 2016 19:21:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533782#comment-15533782

Vrushali C commented on YARN-3981:

Can you give an example of what information is to be written at the flow level? Is it at the
flow level or at flow run level? Put another way, is this information going to be stored each
time say a hive script is run or is it to be written just the very first time it is ever run?
The attributes of a flow run like start time or end time are determined by the coprocessor
automatically so those need not be written specially.

If we need to write information that belongs to a particular flow run but not tied to a specific
application with it, we should write this to the "flow run" table, not the "entity table".

Implementation detail note: the coprocessor is setup for this flow run table, so a little
more attention needed here to ensure we set/do not the right cell tags. 

In order to determine where to have writer processes running and how many and how often, how
to discover those etc, I think it will be helpful to know what kind of information is to be

> support timeline clients not associated with an application
> -----------------------------------------------------------
>                 Key: YARN-3981
>                 URL: https://issues.apache.org/jira/browse/YARN-3981
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Sangjin Lee
>            Assignee: Rohith Sharma K S
>              Labels: YARN-5355
> In the current v.2 design, all timeline writes must belong in a flow/application context
(cluster + user + flow + flow run + application).
> But there are use cases that require writing data outside the context of an application.
One such example is a higher level client (e.g. tez client or hive/oozie/cascading client)
writing flow-level data that spans multiple applications. We need to find a way to support

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message