crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <josh.wi...@gmail.com>
Subject Re: PipelineResult#succeeded interpretation
Date Mon, 19 Jan 2015 18:47:33 GMT
No, that's not good, we should fix that. Is it only in the SparkPipeline
that the situation occurs?

On Mon, Jan 19, 2015 at 8:28 AM, Peter Dolan <peter@nunahealth.com> wrote:

> Hi Crunchers,
>
> At Nuna we've been using Crunch extensively, and I'm really thrilled with
> it.  It's excellent.  There are of course some rough edges though.
>
> Today I ran into some exceptions being thrown in the Spark pipeline, and
> am curious why they weren't resulting in the PipelineResult reporting
> failure.  In particular, my spark pipeline (running with a local spark
> instance, that is with the spark master set to "local[16]") failed with an
> IOException when the machine ran out of space in /tmp/.  The PipelineResult
> retrieved by Pipeline#done returned true from PipelineResult#succeeded.
>
> I've seen this in a couple other contexts, for example when a MapFn threw
> an exception within MapFn#map, which did not result in a false success
> value.
>
> Is this expected / intended behavior?  Should I be getting at the success
> or failure of the execution some other way?
>
> Thanks!
> - Peter
>

Mime
View raw message