Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Received-SPF: pass (nike.apache.org: domain of davidwhiting@gmail.com
 designates 74.125.82.171 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAH29n6O2oqd6ztyDOuq8OU5u5LCHTSNUuOfcyWne91qXxhgdGg@mail.gmail.com>
References: 
 <CAO7kkMPcK00h6-ewyWXGMVtpnp=40n52nqqp7wb7SxLA5+rxZw@mail.gmail.com>
	<CAAZ8DxrJx52N3s9cZ-GQpwMUHNTY+nGFpFogWq3nMourzoPktw@mail.gmail.com>
	<CAH29n6O2oqd6ztyDOuq8OU5u5LCHTSNUuOfcyWne91qXxhgdGg@mail.gmail.com>
Date: Wed, 28 Jan 2015 14:21:34 -0500
Message-ID: 
 <CAGUrFMCdR7DjEAmMO_VUifQvKGKRJu4bjUsEZu5hi=PKsyN9rw@mail.gmail.com>
Subject: Re: .materialize() returns empty collection on pipeline error?
From: David Whiting <davidwhiting@gmail.com>
To: dev@crunch.apache.org
Content-Type: multipart/alternative; boundary=001a11c342ba9b6767050dbb474a

--001a11c342ba9b6767050dbb474a
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I think "fail catastrophically" is probably exactly what should happen
here. You can always catch and use an empty iterable if it fails. A common
use case here is to do one step, materialize it into a collection or map,
then pass that into a DoFn to use as a small lookup table. This failure
mode means that future steps silently continue to execute with empty lookup
tables as part of their processing on the cluster.

On 28 January 2015 at 13:45, Josh Wills <jwills@cloudera.com> wrote:

> Yeah, I think that before, we would just fail catastrophically by throwin=
g
> a CrunchRuntimeException, which I found annoying. Do you prefer that
> behavior? It's certainly something that could be configurable.
>
> J
>
> On Wed, Jan 28, 2015 at 10:36 AM, Jinal Shah <jinalshah2007@gmail.com>
> wrote:
>
> > I think it was intented from these commits I see here
> >
> >
> https://github.com/apache/crunch/commit/3711cea61bded4c90b235a01163ae5f85=
5089917
> > and
> >
> >
> https://github.com/apache/crunch/commit/ded504eb133fa0814e2d90ff2a662e72a=
67e04bb
> > .
> > Josh can enhance on this more.
> >
> > On Wed, Jan 28, 2015 at 9:26 AM, M=C4=81rti=C5=86=C5=A1 Kalv=C4=81ns <
> > martins.kalvans@gmail.com>
> > wrote:
> >
> > > Hi.
> > >
> > > When pipeline fails on cluster with some exception, materialize()
> returns
> > > empty collection and just logs error message.
> > >
> > > I'm (very, very) puzzled about this behaviour:
> > >
> > >
> >
> https://github.com/apache/crunch/blob/master/crunch-core/src/main/java/or=
g/apache/crunch/materialize/MaterializableIterable.java#L92
> > > Is this really intended behaviour?
> > >
> > > If so, then some documentation for materialize() function about this
> > > behaviour would be really nice to have. :)
> > >
> > >
> > > --
> > > M=C4=81rti=C5=86=C5=A1
> > >
> >
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>

--001a11c342ba9b6767050dbb474a--