incubator-crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthias Friedrich <>
Subject Re: Moving PType and friends
Date Sun, 28 Oct 2012 14:46:16 GMT
On Sunday, 2012-10-28, Gabriel Reid wrote:
> On 28 Oct 2012, at 10:39, Matthias Friedrich <> wrote:
> I think that this is exactly what we needed: a clear view on where
> we want to go and what it'll get us (thanks for the outline Matthias).
> Following these principles to remove the violations you listed sounds
> good to me.

Great, thank you!
>> Note: Java 8 will be a game changer; as far as I understand it, our
>> abstract Fns are *not* going to work with Lambdas, we would need
>> interfaces with only a single method. Does anyone think we have to
>> address this before Java 8 is released?
> I don't think we absolutely *need* to address this, but you can be sure that
> I definitely do *want* to address it before the release of Java 8. Seeing the
> way that pipelines can be implemented in Scrunch definitely makes me
> jealous. It looks like we've got about 8 or 9 months to have this sorted out.

The way I see it we have two options: a) change DoFn from an abstract
class to a "functional interface" (that is, an interface that has
exactly one abstract method), or b) provide some kind of mapper that
takes a lambda expression and returns a DoFn.

No matter which way we choose, we need to support both simple Fns that
only implement process() (and are suitable for lambdas) and more
complex Fns that are implemented as traditional Java classes. Complex
Fns cannot be implemented as lambdas because they override
initialize(), cleanup(), etc.

When DoFn is a functional interface, we can work with Java EE style
annotations to tag user-defined methods for cleanup/initialize and use
dependency injection to pass in context.

Big disclaimer: I've read a bit about lambdas but I haven't actually
used them yet. I think we'll have to prototype extensively before
making any decisions on this.


View raw message