crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamal Ananthaneni <kamal.ananthan...@gmail.com>
Subject When can PipelineResult be null?
Date Wed, 31 Dec 2014 04:38:45 GMT
What are the scenarios in which pipeline.run() or pipeline.done() returns a
PipelineResult of null?

We have a pipeline that scans from HBase, does some processing runs the
pipeline and does some additional processing afterwords.

We noticed that if the Scan from HBase failed due to Socket Timeouts or
other exceptions, the PipelineResult from pipeline.run() was null.

We were not treating PipelineResult == null as an error scenario and were
only checking for PipelineResult.status if PipelineResult is != null and
hence the error was not getting propagated to the launcher job.

In this case the error would have propagated and failed our launcher job
had we treated the PipelineResult == null as an Error scenario.

Is the assumption that PipelineResult == null on Pipeline.run() or
Pipeline.done() indicates an error a valid one? In what other scenarios can
a PipelineResult be null?

Thanks,
Kamal

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message