crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Zimmerman (JIRA)" <>
Subject [jira] [Created] (CRUNCH-272) Unable to correlate crunch jobs within Oozie
Date Tue, 01 Oct 2013 22:20:24 GMT
Mike Zimmerman created CRUNCH-272:

             Summary: Unable to correlate crunch jobs within Oozie
                 Key: CRUNCH-272
             Project: Crunch
          Issue Type: Improvement
            Reporter: Mike Zimmerman

I'm not really sure if this should be logged to Oozie or to Crunch, so please feel free to
move as needed.

I would like to request a way to decorate map/reduce jobs that are spawned by a Crunch pipeline
so that I can programmatically determine their origin.  The primary use case for this is integration
with Oozie.  Oozie launches a single map job to run a java action (in our case this java action
runs a crunch job).  Traceability from this original "launcher" job to the jobs created by
the crunch job is impossible without trolling logs.  This leaves a big black hole for the
system operator to assess the performance/impact of these jobs.  My initial thought was to
provide a simple way to indicate a correlationId or similar on a map/reduce job and then make
it accessible within Oozie to query for.  Obviously, that request would have to come after
the correlation feature was available within map/reduce.

This message was sent by Atlassian JIRA

View raw message