hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6027) Support fromId for flows/flowrun apps
Date Wed, 04 Jan 2017 21:24:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15799386#comment-15799386

Sangjin Lee commented on YARN-6027:

bq. At least pagination should be supported for single day flow activities.

OK, I do see your point there. A single day activity having a large number of flows active
is not something that can be ignored, especially on a large/busy cluster. I'm +1 on having
pagination support on a single-day result.

bq. And also I see that flow entities contains all the flow run details. Do we really need
to embed flowruns details in flow entities? Does not it become heavy?

As Varun pointed out, it is not likely there will be a large number of flow runs for a given
flow. Most likely we're dealing with one or a few runs at most. We could add an option of
not returning flow runs if desired, but I think this is not as critical. Having pagination
support may also reduce the need for this, right?

bq. Sangjin Lee, you remember why the user was kept before flow name in row key? To achieve
user level offline aggregation?

If my memory serves right, it was mostly for consistency. A flow is uniquely identified as
a user + flow really. In most tables (the application table might be an exception) we retain
the order of user and flow name. Does that pose a challenge in implementing pagination? We
could consider the fromId as (user + flow), right?

> Support fromId for flows/flowrun apps
> -------------------------------------
>                 Key: YARN-6027
>                 URL: https://issues.apache.org/jira/browse/YARN-6027
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>              Labels: yarn-5355-merge-blocker
> In YARN-5585 , fromId is supported for retrieving entities. We need similar filter for
flows/flowRun apps and flow run and flow as well. 
> Along with supporting fromId, this JIRA should also discuss following points
> * Should we throw an exception for entities/entity retrieval if duplicates found?
> * TimelieEntity :
> ** Should equals method also check for idPrefix?
> ** Does idPrefix is part of identifiers?

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message