crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-538) Add support for Java lambdas to PCollection/PTable methods
Date Tue, 07 Jul 2015 19:34:04 GMT


Gabriel Reid commented on CRUNCH-538:

Being able to use lambdas in Crunch sounds awesome!

I'm curious about what the use case is for the IxxxWithContextFn interfaces is. It seems kind
of unfortunate to put that clear of a mapreduce dependency right in an interface like that
(unless it's really needed of course).

I also noticed that the new PCollection methods aren't overloaded to take a name parameter.
I can get that, as adding an extra overload means double the new methods, but if we don't
have the additional overloads I would almost consider having the single version of those methods
taking a name parameter. On the other hand, that would make the use of lambdas a bit more
noisy again, and would probably be bothersome for people who don't like to use the name parameter
(I try to always use it as it makes debugging so much easier).

Another idea, which might be way off base: what if we made FilterFn implement IFilterFn, and
MapFn implement IMapFn (and possibly something along those lines with IFlatMapFn and DoFn).
I was thinking that it would then be possible to change the parallelDo methods to take the
lambda-style interfaces only, and then either deprecate the existing parallelDo methods or
(even more drastic) remove them completely. I'm thinking that even if we remove the {{parallelDo(DoFn,
PType)}} methods completely, the API would remain backwards compatible (although it would
require recompilation). Am I way off base here, or is that something that could work?

> Add support for Java lambdas to PCollection/PTable methods
> ----------------------------------------------------------
>                 Key: CRUNCH-538
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.12.0
>            Reporter: Josh Wills
>            Assignee: Josh Wills
>             Fix For: 0.13.0
>         Attachments: CRUNCH-538.patch
> Java 8 is more-or-less mainstream at this point, and lambdas are one of its best new
features. Let's add lambda-friendly interfaces and methods to the PCollection/PTable classes
modeled after the methods defined for Scrunch.

This message was sent by Atlassian JIRA

View raw message