crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Kanter (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-272) Unable to correlate crunch jobs within Oozie
Date Fri, 27 Jun 2014 22:27:26 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046476#comment-14046476
] 

Robert Kanter commented on CRUNCH-272:
--------------------------------------

I'd say that it's best if Oozie owns this rather than Crunch.  Otherwise, users have to add
an extra jar to Oozie, add some configs to oozie-site, manually create a "crunch" sharelib,
etc.  If we put it in Oozie, then from the users perspective, this is all built-in and done
for them.

I'll try to take a look early next week.  In the mean time, perhaps you should create an OOZIE
JIRA to "Create a Crunch action"?  

> Unable to correlate crunch jobs within Oozie
> --------------------------------------------
>
>                 Key: CRUNCH-272
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-272
>             Project: Crunch
>          Issue Type: Improvement
>            Reporter: Mike Zimmerman
>            Assignee: Micah Whitacre
>         Attachments: CRUNCH-272.patch, CRUNCH-272_prototype.patch
>
>
> I'm not really sure if this should be logged to Oozie or to Crunch, so please feel free
to move as needed.
> I would like to request a way to decorate map/reduce jobs that are spawned by a Crunch
pipeline so that I can programmatically determine their origin.  The primary use case for
this is integration with Oozie.  Oozie launches a single map job to run a java action (in
our case this java action runs a crunch job).  Traceability from this original "launcher"
job to the jobs created by the crunch job is impossible without trolling logs.  This leaves
a big black hole for the system operator to assess the performance/impact of these jobs. 
My initial thought was to provide a simple way to indicate a correlationId or similar on a
map/reduce job and then make it accessible within Oozie to query for.  Obviously, that request
would have to come after the correlation feature was available within map/reduce.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message