crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Dolan <pe...@nunahealth.com>
Subject PipelineResult#succeeded interpretation
Date Mon, 19 Jan 2015 16:28:42 GMT
Hi Crunchers,

At Nuna we've been using Crunch extensively, and I'm really thrilled with
it.  It's excellent.  There are of course some rough edges though.

Today I ran into some exceptions being thrown in the Spark pipeline, and am
curious why they weren't resulting in the PipelineResult reporting
failure.  In particular, my spark pipeline (running with a local spark
instance, that is with the spark master set to "local[16]") failed with an
IOException when the machine ran out of space in /tmp/.  The PipelineResult
retrieved by Pipeline#done returned true from PipelineResult#succeeded.

I've seen this in a couple other contexts, for example when a MapFn threw
an exception within MapFn#map, which did not result in a false success
value.

Is this expected / intended behavior?  Should I be getting at the success
or failure of the execution some other way?

Thanks!
- Peter

Mime
View raw message