crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: PipelineResult#succeeded interpretation
Date Tue, 20 Jan 2015 17:58:59 GMT
Okay, created https://issues.apache.org/jira/browse/CRUNCH-488 to track it.
Should get a patch together by tmrw.

J

On Mon, Jan 19, 2015 at 4:57 PM, Peter Dolan <peter@nunahealth.com> wrote:

> So far I've only tried this in the SparkPipeline.  In MemPipeline the
> entire JVM dies, so we don't get to determine success or failure.
>
> On Mon, Jan 19, 2015 at 10:47 AM, Josh Wills <josh.wills@gmail.com> wrote:
>
>> No, that's not good, we should fix that. Is it only in the SparkPipeline
>> that the situation occurs?
>>
>> On Mon, Jan 19, 2015 at 8:28 AM, Peter Dolan <peter@nunahealth.com>
>> wrote:
>>
>>> Hi Crunchers,
>>>
>>> At Nuna we've been using Crunch extensively, and I'm really thrilled
>>> with it.  It's excellent.  There are of course some rough edges though.
>>>
>>> Today I ran into some exceptions being thrown in the Spark pipeline, and
>>> am curious why they weren't resulting in the PipelineResult reporting
>>> failure.  In particular, my spark pipeline (running with a local spark
>>> instance, that is with the spark master set to "local[16]") failed with an
>>> IOException when the machine ran out of space in /tmp/.  The PipelineResult
>>> retrieved by Pipeline#done returned true from PipelineResult#succeeded.
>>>
>>> I've seen this in a couple other contexts, for example when a MapFn
>>> threw an exception within MapFn#map, which did not result in a false
>>> success value.
>>>
>>> Is this expected / intended behavior?  Should I be getting at the
>>> success or failure of the execution some other way?
>>>
>>> Thanks!
>>> - Peter
>>>
>>
>>
>


-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
View raw message