crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-538) Add support for Java lambdas to PCollection/PTable methods
Date Thu, 27 Aug 2015 17:45:46 GMT


Gabriel Reid commented on CRUNCH-538:

Oh yeah, I like that way more! This looks really cool.

Something tells me that the nested IDoFn.Context class should be an interface. At the very
least it will hide the update() method, but it just feels like it would be right to use an
interface there too.

I've still got (pretty much) the same remark as before about the lack of a name parameter
for the flatMap, map, and filter methods on PCollection. On the one hand, having an overload
of all of those methods to allow supplying a name parameter seems crazy, and on the other
hand, ensuring that all calls to parallelDo include a name is usually my first step in debugging
a pipeline. Another option would be to only have a single version of these methods that does
include a name parameter, but that also feels like it's going too far. Do you have any strong
feelings on that stuff one way or another?

But anyhow, those nitty details aside, I think this looks pretty awesome.

> Add support for Java lambdas to PCollection/PTable methods
> ----------------------------------------------------------
>                 Key: CRUNCH-538
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.12.0
>            Reporter: Josh Wills
>            Assignee: Josh Wills
>         Attachments: CRUNCH-538.patch, CRUNCH-538b.patch
> Java 8 is more-or-less mainstream at this point, and lambdas are one of its best new
features. Let's add lambda-friendly interfaces and methods to the PCollection/PTable classes
modeled after the methods defined for Scrunch.

This message was sent by Atlassian JIRA

View raw message