hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3588) Timeline entity uniqueness
Date Wed, 25 Jan 2017 18:04:26 GMT

    [ https://issues.apache.org/jira/browse/YARN-3588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838255#comment-15838255

Rohith Sharma K S commented on YARN-3588:

As we are discussed about this  in the weekly call,  few folks asked the question of what
makes entity uniqueness? I believe this is left to the framework publishers as they will be
publishing the entities. Lets say, DAG entities and MR_JOB_ID entities will never be unique
across the application/user/flow/flow-run/clusters. I think we should build a indexing table
for <EntityType, EntityId> that helps a lot while reading entities for given EntityType.
And need not to index all the entities. may be publisher can input that this entity should
be indexed. 

I was looking few existing long running applications where it generates tons of entities which
will be stored in ATSv2. All entities are not important but still they will keep it for future
reference. Only set of entity types are important which are regularly queried. So, these important
entities can be indexed on user demand. 

> Timeline entity uniqueness
> --------------------------
>                 Key: YARN-3588
>                 URL: https://issues.apache.org/jira/browse/YARN-3588
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Zhijie Shen
>            Assignee: Vrushali C
>              Labels: YARN-5355
> In YARN-3051, we have some discussion about how to uniquely identify an entity. Sangjin
and some other folks propose to only uniquely identify an entity by <type, id> in the
scope of a single app. This is different from entity uniqueness in ATSv1, where <type,
id> can globally identify an entity. This is going to affect the way of fetching a single
entity, and raise the compatibility issue. Let's continue our discussion here to unblock YARN-3051.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message