hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2928) Application Timeline Server (ATS) next gen: phase 1
Date Fri, 06 Feb 2015 22:28:37 GMT

    [ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310056#comment-14310056

Zhijie Shen commented on YARN-2928:

bq. A single tez application can run multiple different Hive queries submitted by different

In this use case, who is the user of the TEZ application? This may affect the data mode and
the parent-child relationship (cluster->user->flow->flow run->application).

bq. Where does the current implementation's "otherInfo" and "primaryFilters" fit in? 

"metadata" aims to store the same thing as "otherInfo", but I didn't want to be called "otherinfo"
because it's no longer the other info than "primaryFilters". When making the new schema, I'm
looking for the option to have the entity indexed, but don't need to explicitly specify what
is the "primaryFilters", which makes trouble and bugs when updating the entity before.

bq. What are the main differences between meta-data and configuration? 

It may be combined, as I consider both are key-value pairs, but I distinguish them explicitly
for better usage. Or is there any special access pattern for config?

bq. If there is a hierarchy of objects, will there be support to listen to or retrieve all
events for a given tree by providing a root node?

We may probably run adhoc query to get the events of all applications of a workflow.

bq. What use are events? Will there be a "streaming" API available to listen to all events
based on some search criteria?
bq. In certain cases, it might be required to mine a specific job's data by exporting contents
out of ATS.

They sound to be interesting features, but we may not able to accommodate them within Hadoop
2.8 timeline.

> Application Timeline Server (ATS) next gen: phase 1
> ---------------------------------------------------
>                 Key: YARN-2928
>                 URL: https://issues.apache.org/jira/browse/YARN-2928
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Priority: Critical
>         Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, Data model proposal v1.pdf
> We have the application timeline server implemented in yarn per YARN-1530 and YARN-321.
Although it is a great feature, we have recognized several critical issues and features that
need to be addressed.
> This JIRA proposes the design and implementation changes to address those. This is phase
1 of this effort.

This message was sent by Atlassian JIRA

View raw message