crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamal Ananthaneni <kamal.ananthan...@gmail.com>
Subject Re: When can PipelineResult be null?
Date Wed, 31 Dec 2014 14:52:52 GMT
Thanks Jinal and Gabriel for the prompt response!

Since crunch-443 is not yet resolved, we will make the change on our side
to handle pipelineResult == null as a failure.

Thanks,
Kamal

On Wed, Dec 31, 2014 at 3:48 AM, Gabriel Reid <gabriel.reid@gmail.com>
wrote:

> I see the returning of null for a PipelineResult as a bug in Crunch,
> and logged CRUNCH-443 [1] a while back (although it's still open).
> It's not totally clear what the best thing to do would be in such a
> situation, but I think that the returning of null should definitely
> change.
>
> - Gabriel
>
>
> 1. https://issues.apache.org/jira/browse/CRUNCH-443
>
> On Wed, Dec 31, 2014 at 6:55 AM, Jinal Shah <jinalshah2007@gmail.com>
> wrote:
> > Hi Kamal,
> >
> > PipelineResult can be null. This will happen when there is a job failure
> in
> > the pipeline(in almost all the cases I have seen). Not propagating to the
> > launcher is because internally Crunch handles any exceptions and return a
> > NULL object instead of throwing the exception. You could get the
> exception
> > if you look at jobs details page. You could possibly raise the argument
> to
> > throw an exception when a jobs from pipeline.run and pipeline.done.
> >
> > Thanks,
> > Jinal
> >
> > On Tue, Dec 30, 2014 at 8:38 PM, Kamal Ananthaneni <
> > kamal.ananthaneni@gmail.com> wrote:
> >
> >> What are the scenarios in which pipeline.run() or pipeline.done()
> returns a
> >> PipelineResult of null?
> >>
> >> We have a pipeline that scans from HBase, does some processing runs the
> >> pipeline and does some additional processing afterwords.
> >>
> >> We noticed that if the Scan from HBase failed due to Socket Timeouts or
> >> other exceptions, the PipelineResult from pipeline.run() was null.
> >>
> >> We were not treating PipelineResult == null as an error scenario and
> were
> >> only checking for PipelineResult.status if PipelineResult is != null and
> >> hence the error was not getting propagated to the launcher job.
> >>
> >> In this case the error would have propagated and failed our launcher job
> >> had we treated the PipelineResult == null as an Error scenario.
> >>
> >> Is the assumption that PipelineResult == null on Pipeline.run() or
> >> Pipeline.done() indicates an error a valid one? In what other scenarios
> can
> >> a PipelineResult be null?
> >>
> >> Thanks,
> >> Kamal
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message