hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hitesh Shah (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1390) Provide a way to capture source of an application to be queried through REST or Java Client APIs
Date Tue, 12 Nov 2013 10:35:19 GMT

    [ https://issues.apache.org/jira/browse/YARN-1390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819992#comment-13819992

Hitesh Shah commented on YARN-1390:

Also, from an implementation point of view, I would assume this map would be *not* be searchable.
Free-form text or even a set of variable key-val pairs are expensive to search. Only defined
fields such as applicationType ( which would contain only a single value ) should be searchable.

bq. Representing applicationType as a set should suffice.

Representing it as a set is fine. However, how do you expect Oozie to pass source info to
Pig which in turn will pass it to MR ? Are you assuming the source info is just a simple well
defined string such as "Oozie" or would Oozie do something like "Oozie:workflowId=1234" ?
I think lineage is something which YARN does not need to know or understand at the moment.
Better to support it via the free-form map instead of introducing a new field which we are
not sure how we plan to use/handle/support. 

> Provide a way to capture source of an application to be queried through REST or Java
Client APIs
> ------------------------------------------------------------------------------------------------
>                 Key: YARN-1390
>                 URL: https://issues.apache.org/jira/browse/YARN-1390
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: api
>    Affects Versions: 2.2.0
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
> In addition to other fields like application-type (added in YARN-563), it is useful to
have an applicationSource field to track the source of an application. The application source
can be useful in (1) fetching only those applications a user is interested in, (2) potentially
adding source-specific optimizations in the future. 
> Examples of sources are: User-defined project names, Pig, Hive, Oozie, Sqoop etc.

This message was sent by Atlassian JIRA

View raw message