crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabriel Reid <gabriel.r...@gmail.com>
Subject Re: When can PipelineResult be null?
Date Wed, 31 Dec 2014 09:48:34 GMT
I see the returning of null for a PipelineResult as a bug in Crunch,
and logged CRUNCH-443 [1] a while back (although it's still open).
It's not totally clear what the best thing to do would be in such a
situation, but I think that the returning of null should definitely
change.

- Gabriel


1. https://issues.apache.org/jira/browse/CRUNCH-443

On Wed, Dec 31, 2014 at 6:55 AM, Jinal Shah <jinalshah2007@gmail.com> wrote:
> Hi Kamal,
>
> PipelineResult can be null. This will happen when there is a job failure in
> the pipeline(in almost all the cases I have seen). Not propagating to the
> launcher is because internally Crunch handles any exceptions and return a
> NULL object instead of throwing the exception. You could get the exception
> if you look at jobs details page. You could possibly raise the argument to
> throw an exception when a jobs from pipeline.run and pipeline.done.
>
> Thanks,
> Jinal
>
> On Tue, Dec 30, 2014 at 8:38 PM, Kamal Ananthaneni <
> kamal.ananthaneni@gmail.com> wrote:
>
>> What are the scenarios in which pipeline.run() or pipeline.done() returns a
>> PipelineResult of null?
>>
>> We have a pipeline that scans from HBase, does some processing runs the
>> pipeline and does some additional processing afterwords.
>>
>> We noticed that if the Scan from HBase failed due to Socket Timeouts or
>> other exceptions, the PipelineResult from pipeline.run() was null.
>>
>> We were not treating PipelineResult == null as an error scenario and were
>> only checking for PipelineResult.status if PipelineResult is != null and
>> hence the error was not getting propagated to the launcher job.
>>
>> In this case the error would have propagated and failed our launcher job
>> had we treated the PipelineResult == null as an Error scenario.
>>
>> Is the assumption that PipelineResult == null on Pipeline.run() or
>> Pipeline.done() indicates an error a valid one? In what other scenarios can
>> a PipelineResult be null?
>>
>> Thanks,
>> Kamal
>>

Mime
View raw message