crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <josh.wi...@gmail.com>
Subject Re: testing if a pipeline was successful
Date Sat, 08 Jun 2013 15:46:58 GMT
Yeah, I'm not sure how I ever thought that would work. If you have a patch,
let's get it in.

J


On Fri, Jun 7, 2013 at 9:42 PM, Joseph Adler <joseph.adler@gmail.com> wrote:

> Thanks Josh.
>
> Let me know what you think. I have a patch for this (so pipelines don't
> succeed if one of the stages fails).
>
> -- Joe
>
>
> On Thu, Jun 6, 2013 at 1:49 PM, Josh Wills <jwills@cloudera.com> wrote:
>
>> Definitely sounds like a bug. Will take a look when I get on this plane.
>> On Jun 6, 2013 12:36 PM, "Joseph Adler" <joseph.adler@gmail.com> wrote:
>>
>>> Hi guys,
>>>
>>> I'm confused by how to tell if a Crunch Pipeline has succeeded. I
>>> suspect that the current implementation is not correct, but want to verify
>>> that before I file a Jira and submit a patch.
>>>
>>> Here's what I'm trying to do: I'd like to wrap a crunch job inside
>>> another Java program (so that I can execute it in a larger
>>> scheduling/workflow tool like Azkaban or Oozie). So, I'd like to build and
>>> execute a Crunch pipeline, then test if it succeeded.
>>>
>>> I'd like to do something like:
>>>
>>>     PipelineResult result = pipeline.run();
>>>     if (!result.succeeded())
>>>     {
>>>         throw new Exception("Job failed: " + result.toString() + "\n");
>>>     }
>>>     return 0;
>>>
>>>
>>> Unfortunately, this doesn't work. Looking at the implementation of
>>> PipelineResult, here is how the succeeded method is implemented:
>>>
>>>   public boolean succeeded() {
>>>     return !stageResults.isEmpty();
>>>   }
>>>
>>> Looking at where the PipelineResult object is created (in MRExecutor),
>>> it looks like the list of stage results will be non-empty regardless of
>>> whether the job succeeded or not:
>>>
>>>        ...
>>>        result = new PipelineResult(stages);
>>>         if (killSignal.getCount() != 0) {
>>>           status.set(Status.KILLED);
>>>         } else {
>>>           status.set(result.succeeded() ? Status.SUCCEEDED :
>>> Status.FAILED);
>>>         }
>>>         ...
>>>
>>> So, even if one of the stages fails, the result of executing the
>>> pipeline will be "succeeded."
>>>
>>> I can't find an alternative way to test if a failure occurred during a
>>> pipeline execution. Is there something that I'm missing, or is the
>>> implementation incorrect?
>>>
>>> -- Joe
>>>
>>
>

Mime
View raw message