hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hitesh Shah (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1390) Provide a way to capture source of an application to be queried through REST or Java Client APIs
Date Mon, 11 Nov 2013 10:59:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-1390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13818863#comment-13818863

Hitesh Shah commented on YARN-1390:

[~vinodkv] How is the application going to be identified from a application history point
of view? 

There seems to be 2 different things which are required. Lineage to understand how an application
was submitted ( this could be multi-levels deep ) and the other to identify the application
itself. For example, what is the plan for a oozie job that launches a pig script that in turn
runs multiple mapreduce jobs? 

I think applicationType as  it stands today should not change and should remain hardcoded
by MR. For lineage, something else should be introduced but it requires each and every layer
to cooperate to augment the lineage data. I dont think there is a quick fix here. This is
something which can be introduced at the hadoop layer but will need to traverse through the
whole ecosystem for it to work correctly.

> Provide a way to capture source of an application to be queried through REST or Java
Client APIs
> ------------------------------------------------------------------------------------------------
>                 Key: YARN-1390
>                 URL: https://issues.apache.org/jira/browse/YARN-1390
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: api
>    Affects Versions: 2.2.0
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
> In addition to other fields like application-type (added in YARN-563), it is useful to
have an applicationSource field to track the source of an application. The application source
can be useful in (1) fetching only those applications a user is interested in, (2) potentially
adding source-specific optimizations in the future. 
> Examples of sources are: User-defined project names, Pig, Hive, Oozie, Sqoop etc.

This message was sent by Atlassian JIRA

View raw message