crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-272) Unable to correlate crunch jobs within Oozie
Date Sun, 13 Apr 2014 23:30:15 GMT


Micah Whitacre updated CRUNCH-272:

    Attachment: CRUNCH-272_prototype.patch

So in thinking about how to solve this I see about 2-3 solutions:

1. We create a custom Oozie action that does a similar action to what the HiveAction does[1][2]
and simply search the log files for strings that match a regex.  This is was called suboptimal
by Oozie community as I suggested they make a small tweak to JavaAction to do this and they
weren't enthused by it.  Additionally we'd have to pic a regex that might fit all of our pipeline
type jobs (MR + Spark).
2. We create a launcher framework and most likely a Launcher API that all Crunch consumers
wanting to use Oozie would have to implement so that they could report back to us their PipelineResults
and we pull the job ids off of that.  This would then be coupled with a custom Oozie Action.
3. We continue to let consumers utilize the standard Oozie Java action and instead provide
facilities/helpers to report the child job ids in a consistent manner with how Oozie expects
the job ids to be reported.  This is exactly what is contained inside this patch.  The downside
with this approach is that if jobs that were invoked due to a materialize/iterator call are
not tracked in the PipelineResult stages we would be missing some jobs.  This would be a problem
with approach #2 as well.

I still need to test out may prototype code on a cluster with Oozie but I'm reasonably sure
it will work.  Thoughts?

[1] -
[2] -

> Unable to correlate crunch jobs within Oozie
> --------------------------------------------
>                 Key: CRUNCH-272
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>            Reporter: Mike Zimmerman
>            Assignee: Micah Whitacre
>         Attachments: CRUNCH-272_prototype.patch
> I'm not really sure if this should be logged to Oozie or to Crunch, so please feel free
to move as needed.
> I would like to request a way to decorate map/reduce jobs that are spawned by a Crunch
pipeline so that I can programmatically determine their origin.  The primary use case for
this is integration with Oozie.  Oozie launches a single map job to run a java action (in
our case this java action runs a crunch job).  Traceability from this original "launcher"
job to the jobs created by the crunch job is impossible without trolling logs.  This leaves
a big black hole for the system operator to assess the performance/impact of these jobs. 
My initial thought was to provide a simple way to indicate a correlationId or similar on a
map/reduce job and then make it accessible within Oozie to query for.  Obviously, that request
would have to come after the correlation feature was available within map/reduce.

This message was sent by Atlassian JIRA

View raw message