crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Dolan <pe...@nunahealth.com>
Subject Re: PipelineResult#succeeded interpretation
Date Tue, 20 Jan 2015 00:57:12 GMT
So far I've only tried this in the SparkPipeline.  In MemPipeline the
entire JVM dies, so we don't get to determine success or failure.

On Mon, Jan 19, 2015 at 10:47 AM, Josh Wills <josh.wills@gmail.com> wrote:

> No, that's not good, we should fix that. Is it only in the SparkPipeline
> that the situation occurs?
>
> On Mon, Jan 19, 2015 at 8:28 AM, Peter Dolan <peter@nunahealth.com> wrote:
>
>> Hi Crunchers,
>>
>> At Nuna we've been using Crunch extensively, and I'm really thrilled with
>> it.  It's excellent.  There are of course some rough edges though.
>>
>> Today I ran into some exceptions being thrown in the Spark pipeline, and
>> am curious why they weren't resulting in the PipelineResult reporting
>> failure.  In particular, my spark pipeline (running with a local spark
>> instance, that is with the spark master set to "local[16]") failed with an
>> IOException when the machine ran out of space in /tmp/.  The PipelineResult
>> retrieved by Pipeline#done returned true from PipelineResult#succeeded.
>>
>> I've seen this in a couple other contexts, for example when a MapFn threw
>> an exception within MapFn#map, which did not result in a false success
>> value.
>>
>> Is this expected / intended behavior?  Should I be getting at the success
>> or failure of the execution some other way?
>>
>> Thanks!
>> - Peter
>>
>
>

Mime
View raw message