hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vrushali C (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6733) Add table for storing sub-application entities
Date Mon, 24 Jul 2017 22:36:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-6733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099207#comment-16099207

Vrushali C commented on YARN-6733:

So we thought that it will be good to keep the column name so that sub apps can store this
information. For regular applications, the flow version can be used to determine whether optimizations
are to be done. The flow version indicates if the flow has changed, that is, say if the pig
script changes, it's flow version will change. So then, for example, reducer estimation calculations
can be done differently. This applies to the application entities. We discussed that it will
be good to keep the same information for sub-apps in case they want to use this information
in a similar fashion. As such, this column currently only exists in code, it's not taking
up any disk space/hbase space etc if no one writes to it. But having it given the framework
developers a chance to use it if they want. 

> Add table for storing sub-application entities
> ----------------------------------------------
>                 Key: YARN-6733
>                 URL: https://issues.apache.org/jira/browse/YARN-6733
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Vrushali C
>            Assignee: Vrushali C
>         Attachments: IMG_7040.JPG, YARN-6733-YARN-5355.001.patch, YARN-6733-YARN-5355.002.patch,
YARN-6733-YARN-5355.003.patch, YARN-6733-YARN-5355.004.patch, YARN-6733-YARN-5355.005.patch,
YARN-6733-YARN-5355.006.patch, YARN-6733-YARN-5355.007.patch, YARN-6733-YARN-5355.008.patch
> After a discussion with Tez folks, we have been thinking over introducing a table to
store  sub-application information.
> For example, if a Tez session runs for a certain period as User X and runs a few AMs.
These AMs accept DAGs from other users. Tez will execute these dags with a doAs user. ATSv2
should store this information in a new table perhaps called as "sub_application" table. 
> This jira tracks the code changes needed for  table schema creation.
> I will file other jiras for writing to that table, updating the user name fields to include
sub-application user etc.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message