Return-Path: 
 <dev-return-5885-apmail-crunch-dev-archive=crunch.apache.org@crunch.apache.org>
X-Original-To: apmail-crunch-dev-archive@www.apache.org
Delivered-To: apmail-crunch-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 65CB817DF0
	for <apmail-crunch-dev-archive@www.apache.org>;
 Sat,  9 Jan 2016 22:11:40 +0000 (UTC)
Received: (qmail 90502 invoked by uid 500); 9 Jan 2016 22:11:40 -0000
Delivered-To: apmail-crunch-dev-archive@crunch.apache.org
Received: (qmail 90457 invoked by uid 500); 9 Jan 2016 22:11:40 -0000
Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@crunch.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@crunch.apache.org>
List-Post: <mailto:dev@crunch.apache.org>
List-Id: <dev.crunch.apache.org>
Reply-To: dev@crunch.apache.org
Delivered-To: mailing list dev@crunch.apache.org
Received: (qmail 90445 invoked by uid 500); 9 Jan 2016 22:11:40 -0000
Delivered-To: apmail-incubator-crunch-dev@incubator.apache.org
Received: (qmail 90438 invoked by uid 99); 9 Jan 2016 22:11:40 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 09 Jan 2016 22:11:40 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id C39C02C14F4
	for <crunch-dev@incubator.apache.org>; Sat,  9 Jan 2016 22:11:39 +0000 (UTC)
Date: Sat, 9 Jan 2016 22:11:39 +0000 (UTC)
From: "David Whiting (JIRA)" <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <JIRA.12924394.1451208725000.66909.1452377499798@Atlassian.JIRA>
In-Reply-To: <JIRA.12924394.1451208725000@Atlassian.JIRA>
References: <JIRA.12924394.1451208725000@Atlassian.JIRA>
 <JIRA.12924394.1451208725237@arcas>
Subject: [jira] [Commented] (CRUNCH-585) Move Java 8 lambda support into
 separate module
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/CRUNCH-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15090802#comment-15090802 ] 

David Whiting commented on CRUNCH-585:
--------------------------------------

I think it's possible to set up a profile for java 8 conditional on jdk version >=1.8, then add the additional crunch-lambda module as part of the profile definition. I tried this:

  <profiles>
    <profile>
      <id>java-8</id>
      <activation>
        <jdk>[1.8,]</jdk>
      </activation>
      <modules>
        <module>crunch-lambda</module>
      </modules>
    </profile>
  </profiles>

and it seems to compile everything as expected, but I don't have a <=1.7 jdk to make sure it doesn't try and compile the module on that.

I can add this change to the next patch version (also with some minor fixes and better testing and documentation). 

> Move Java 8 lambda support into separate module 
> ------------------------------------------------
>
>                 Key: CRUNCH-585
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-585
>             Project: Crunch
>          Issue Type: Improvement
>            Reporter: David Whiting
>             Fix For: 0.14.0
>
>         Attachments: 0001-Java-8-lambda-support-for-Apache-Crunch.patch, 0001-Java-8-lambda-support-for-Apache-Crunch.patch
>
>
> As discussed on a previous dev list thread, this patch implements a set of operations to conveniently use Java 8 lambda expressions and method references to construct Crunch pipelines by wrapping the PCollection instances into analogous "LCollection" instances which delegate the necessary operations, in much the same way the Scrunch wraps the Crunch Core API.
> I'm still not 100% convinced that this is better for the user than the existing lambda support via IMapFn and IDoFn PCollection operations, so I'm still interested in people's views on this.
> Advantages:
> - Concise self-contained implementation
> - Methods implemented in terms of a very basic subset of PCollection operations (useful if we want to scale down the PCollection API at some point)
> - API can be written in terms of the Java 8 library, operating on streams and functional interfaces, making in more familiar to a new developer.
> - Retain "type '.' and see what I can do" experience.
> - Really easy to add new operations (just default method on interface)
> Disadvantages:
> - PCollections must be wrapped into LCollections before use.
> - LCollections must be unwrapped into PCollections to access some existing operations.
> - Using counters and other contextual data is far more complex.
> Some limitations of this particular patch:
> - Some omissions in API (not sure how much to implement)
> - No Javadocs yet.
> - Very poor tests.
> - Naming is a bit off (eg. reduce() or reduceValues(), get() or underlying())
> I can fix all that, but I wanted to bring the community in at this point to get some feedback on both the idea and the implementation as it's quite a big patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)