crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-272) Unable to correlate crunch jobs within Oozie
Date Tue, 01 Apr 2014 01:13:15 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13955974#comment-13955974
] 

Micah Whitacre commented on CRUNCH-272:
---------------------------------------

So I spent some time last week poking through Oozie code trying to figure out how Oozie was
doing it for other processes like Hive.  It seems like how they solved the problem for the
Hive action[1] could be generically applied to the Java action in general.  Therefore i logged
an enhancement to Oozie to possibly solve this.[2]

[1] - https://github.com/apache/oozie/blob/master/sharelib/hive/src/main/java/org/apache/oozie/action/hadoop/HiveMain.java#L295
[2] - https://issues.apache.org/jira/browse/OOZIE-1767

> Unable to correlate crunch jobs within Oozie
> --------------------------------------------
>
>                 Key: CRUNCH-272
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-272
>             Project: Crunch
>          Issue Type: Improvement
>            Reporter: Mike Zimmerman
>
> I'm not really sure if this should be logged to Oozie or to Crunch, so please feel free
to move as needed.
> I would like to request a way to decorate map/reduce jobs that are spawned by a Crunch
pipeline so that I can programmatically determine their origin.  The primary use case for
this is integration with Oozie.  Oozie launches a single map job to run a java action (in
our case this java action runs a crunch job).  Traceability from this original "launcher"
job to the jobs created by the crunch job is impossible without trolling logs.  This leaves
a big black hole for the system operator to assess the performance/impact of these jobs. 
My initial thought was to provide a simple way to indicate a correlationId or similar on a
map/reduce job and then make it accessible within Oozie to query for.  Obviously, that request
would have to come after the correlation feature was available within map/reduce.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message