falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FALCON-510) Inject falcon related properties to job conf
Date Thu, 02 Oct 2014 01:54:34 GMT

    [ https://issues.apache.org/jira/browse/FALCON-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155986#comment-14155986
] 

Srikanth Sundarrajan edited comment on FALCON-510 at 10/2/14 1:54 AM:
----------------------------------------------------------------------

[~peeyushb], We ought to add it in such a way that this would work for actions in user workflow
as well as the pig scripts or hive launched from within user workflow actions. Would urge
you to look at this specific [oozie feature|http://archive.cloudera.com/cdh4/cdh/4/oozie/WorkflowFunctionalSpec.html#a19_Global_Configurations]
to enable this.


was (Author: sriksun):
[~peeyushb], We ought to add it in such a way that this work for actions in user workflow
as well as the pig scripts or hive launched from within user workflow actions. Would urge
you to look at this specific [oozie feature|http://archive.cloudera.com/cdh4/cdh/4/oozie/WorkflowFunctionalSpec.html#a19_Global_Configurations]
to enable this.

> Inject falcon related properties to job conf
> --------------------------------------------
>
>                 Key: FALCON-510
>                 URL: https://issues.apache.org/jira/browse/FALCON-510
>             Project: Falcon
>          Issue Type: Improvement
>            Reporter: Shwetha G S
>            Assignee: Peeyush Bishnoi
>         Attachments: falcon-510.txt
>
>
> Currently there is no falcon context injected at MR job level. The job conf has at most
the oozie workflow / action ID either in the job name or sometimes in the job conf.
> Therefore there is no way for a tool like hraven, which relies completely on jobconf
and job history data, to identify that a particular job maps to a particular falcon process
or it's instance time, etc. Right now hraven does regex-based job name surgery on a best effort
basis before emitting metrics to graphite
> Request the following feature in falcon:
> Add the following properties to the job conf (for all jobs - be it a pig action or an
MR action):
> falcon.process.name
> falcon.process.instancetime
> while we're at it, might as well add any other falcon context as a jobconf property (like
whether it was a rerun or the input/output feeds, cluster, validity, any process properties,
etc.)
> This will ofcourse inject at the first job level and cannot ensure that any child jobs
get the properties passed on (unless we can figure out a way to do that too).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message