crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jinal Shah <jinalshah2...@gmail.com>
Subject Re: When can PipelineResult be null?
Date Wed, 31 Dec 2014 05:55:35 GMT
Hi Kamal,

PipelineResult can be null. This will happen when there is a job failure in
the pipeline(in almost all the cases I have seen). Not propagating to the
launcher is because internally Crunch handles any exceptions and return a
NULL object instead of throwing the exception. You could get the exception
if you look at jobs details page. You could possibly raise the argument to
throw an exception when a jobs from pipeline.run and pipeline.done.

Thanks,
Jinal

On Tue, Dec 30, 2014 at 8:38 PM, Kamal Ananthaneni <
kamal.ananthaneni@gmail.com> wrote:

> What are the scenarios in which pipeline.run() or pipeline.done() returns a
> PipelineResult of null?
>
> We have a pipeline that scans from HBase, does some processing runs the
> pipeline and does some additional processing afterwords.
>
> We noticed that if the Scan from HBase failed due to Socket Timeouts or
> other exceptions, the PipelineResult from pipeline.run() was null.
>
> We were not treating PipelineResult == null as an error scenario and were
> only checking for PipelineResult.status if PipelineResult is != null and
> hence the error was not getting propagated to the launcher job.
>
> In this case the error would have propagated and failed our launcher job
> had we treated the PipelineResult == null as an Error scenario.
>
> Is the assumption that PipelineResult == null on Pipeline.run() or
> Pipeline.done() indicates an error a valid one? In what other scenarios can
> a PipelineResult be null?
>
> Thanks,
> Kamal
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message