crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Dolan <pe...@nunahealth.com>
Subject Re: PipelineResult#succeeded interpretation
Date Tue, 20 Jan 2015 18:01:28 GMT
Thanks Josh!

On Tue, Jan 20, 2015 at 9:58 AM, Josh Wills <jwills@cloudera.com> wrote:

> Okay, created https://issues.apache.org/jira/browse/CRUNCH-488 to track
> it. Should get a patch together by tmrw.
>
> J
>
> On Mon, Jan 19, 2015 at 4:57 PM, Peter Dolan <peter@nunahealth.com> wrote:
>
>> So far I've only tried this in the SparkPipeline.  In MemPipeline the
>> entire JVM dies, so we don't get to determine success or failure.
>>
>> On Mon, Jan 19, 2015 at 10:47 AM, Josh Wills <josh.wills@gmail.com>
>> wrote:
>>
>>> No, that's not good, we should fix that. Is it only in the SparkPipeline
>>> that the situation occurs?
>>>
>>> On Mon, Jan 19, 2015 at 8:28 AM, Peter Dolan <peter@nunahealth.com>
>>> wrote:
>>>
>>>> Hi Crunchers,
>>>>
>>>> At Nuna we've been using Crunch extensively, and I'm really thrilled
>>>> with it.  It's excellent.  There are of course some rough edges though.
>>>>
>>>> Today I ran into some exceptions being thrown in the Spark pipeline,
>>>> and am curious why they weren't resulting in the PipelineResult reporting
>>>> failure.  In particular, my spark pipeline (running with a local spark
>>>> instance, that is with the spark master set to "local[16]") failed with an
>>>> IOException when the machine ran out of space in /tmp/.  The PipelineResult
>>>> retrieved by Pipeline#done returned true from PipelineResult#succeeded.
>>>>
>>>> I've seen this in a couple other contexts, for example when a MapFn
>>>> threw an exception within MapFn#map, which did not result in a false
>>>> success value.
>>>>
>>>> Is this expected / intended behavior?  Should I be getting at the
>>>> success or failure of the execution some other way?
>>>>
>>>> Thanks!
>>>> - Peter
>>>>
>>>
>>>
>>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>

Mime
View raw message