crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joseph Adler <joseph.ad...@gmail.com>
Subject Re: testing if a pipeline was successful
Date Sat, 08 Jun 2013 04:42:57 GMT
Thanks Josh.

Let me know what you think. I have a patch for this (so pipelines don't
succeed if one of the stages fails).

-- Joe


On Thu, Jun 6, 2013 at 1:49 PM, Josh Wills <jwills@cloudera.com> wrote:

> Definitely sounds like a bug. Will take a look when I get on this plane.
> On Jun 6, 2013 12:36 PM, "Joseph Adler" <joseph.adler@gmail.com> wrote:
>
>> Hi guys,
>>
>> I'm confused by how to tell if a Crunch Pipeline has succeeded. I suspect
>> that the current implementation is not correct, but want to verify that
>> before I file a Jira and submit a patch.
>>
>> Here's what I'm trying to do: I'd like to wrap a crunch job inside
>> another Java program (so that I can execute it in a larger
>> scheduling/workflow tool like Azkaban or Oozie). So, I'd like to build and
>> execute a Crunch pipeline, then test if it succeeded.
>>
>> I'd like to do something like:
>>
>>     PipelineResult result = pipeline.run();
>>     if (!result.succeeded())
>>     {
>>         throw new Exception("Job failed: " + result.toString() + "\n");
>>     }
>>     return 0;
>>
>>
>> Unfortunately, this doesn't work. Looking at the implementation of
>> PipelineResult, here is how the succeeded method is implemented:
>>
>>   public boolean succeeded() {
>>     return !stageResults.isEmpty();
>>   }
>>
>> Looking at where the PipelineResult object is created (in MRExecutor), it
>> looks like the list of stage results will be non-empty regardless of
>> whether the job succeeded or not:
>>
>>        ...
>>        result = new PipelineResult(stages);
>>         if (killSignal.getCount() != 0) {
>>           status.set(Status.KILLED);
>>         } else {
>>           status.set(result.succeeded() ? Status.SUCCEEDED :
>> Status.FAILED);
>>         }
>>         ...
>>
>> So, even if one of the stages fails, the result of executing the pipeline
>> will be "succeeded."
>>
>> I can't find an alternative way to test if a failure occurred during a
>> pipeline execution. Is there something that I'm missing, or is the
>> implementation incorrect?
>>
>> -- Joe
>>
>

Mime
View raw message