From dev-return-16443-archive-asf-public=cust-asf.ponee.io@nifi.apache.org Tue Feb 13 18:06:22 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 0AD72180656 for ; Tue, 13 Feb 2018 18:06:20 +0100 (CET) Received: (qmail 68848 invoked by uid 500); 13 Feb 2018 17:06:19 -0000 Mailing-List: contact dev-help@nifi.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nifi.apache.org Delivered-To: mailing list dev@nifi.apache.org Received: (qmail 68824 invoked by uid 99); 13 Feb 2018 17:06:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Feb 2018 17:06:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id B2565C0118 for ; Tue, 13 Feb 2018 17:06:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id b2W7CRmf60Ke for ; Tue, 13 Feb 2018 17:06:14 +0000 (UTC) Received: from mail-oi0-f45.google.com (mail-oi0-f45.google.com [209.85.218.45]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 098BB5F180 for ; Tue, 13 Feb 2018 17:06:13 +0000 (UTC) Received: by mail-oi0-f45.google.com with SMTP id u6so14397960oiv.9 for ; Tue, 13 Feb 2018 09:06:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=lwhAeiSxXcAi9Lq0FjN6f+tZKnX4TwfJvI27Jn4yRrg=; b=OFplmor0pa7KsHqVfOau6z88HWcfd252NcSYEBJxKGCCxwO+m3AdMtEcjZhHaK7E3v oW4gqmXeX0uAVE6vET1qgXsmjiAc+qf68gsYDP0DflZgi8OCRYiFNuyiIAcGQ/4sbSpo qURMbwwpxREsWmUDonkTZNdF1da4i7LS0RYQ//kVk5N0bVmFiotieWggGdV520504nNH /7eB58f+O8w3LinP46DcMNHgaLCRRvhIFEDd+FEAJSEXD0Dig35djXw5Cr74hjY6gFwi 9dSQOxMBhz7cio1nG68jfxyED4giffAcWr+yhWFSGliWlpaVvK0z68dg7d9ijQCwTtYM KuQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=lwhAeiSxXcAi9Lq0FjN6f+tZKnX4TwfJvI27Jn4yRrg=; b=T+nuD5UXxEQH3fR00QWvp2p/vX08YSs7C4dE6B1uyS0owgQfsF+WGN0lUzcq256FOW +RCTWnikdfA2mpGHTJA5mwB/hvREY2yJ7qviiiz4ANZ+lS8FzhIoJV5lpn4tKaCyJHfh P9KymHNjb3ZaYquXM/TGX8UDF3uJNbcWs4RYNpd/qIWWI+Rgkh+pomhNpOejAeIRtnAJ y/KD1Il3sZhbEX250mO3am7VuAXLf+mHxd3pzmL6V453mRpqhYJRq3tY6/dCEl8KHXLb 4O9dDCOf4jBIJUSwZRwn1QEdRufadCP+huaVCiPVerIdUe1xQpV3yk/N8hz569iqUT+J TnIg== X-Gm-Message-State: APf1xPDrUhc/NzzcwnylS/12yk1JOlisYjYDQEQ5rQaklg028zPi/8yV LYa2xebRPLP6tiMYA/V8pAWJAmFDAqTFgEwpDfk= X-Google-Smtp-Source: AH8x22519T8QDsv18k/aVWxlob5vvQkBb6genpaK8Rp/3e5ADhwaVqna+A7hhxtQEgA/DP/xOugdwatbezEBSvU2DxE= X-Received: by 10.202.72.6 with SMTP id v6mr1367086oia.237.1518541571333; Tue, 13 Feb 2018 09:06:11 -0800 (PST) MIME-Version: 1.0 Received: by 10.74.187.22 with HTTP; Tue, 13 Feb 2018 09:06:10 -0800 (PST) In-Reply-To: References: From: Mike Thomsen Date: Tue, 13 Feb 2018 12:06:10 -0500 Message-ID: Subject: Re: Will you accept contributions in Scala? To: dev@nifi.apache.org Content-Type: multipart/alternative; boundary="001a113db248efa42405651b02ed" --001a113db248efa42405651b02ed Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Milan, I don't think you can do that without creating a lot of fuel for a flame war. I have personally never met a Scala developer who was incapable of writing decent Java. The same is true of Groovy. I think anyone who finds the requirement to use Java over their language of choice to be a deal breaker on contributing is probably someone unlikely to be more help than trouble anyway. Mike On Tue, Feb 13, 2018 at 11:35 AM, Milan Das wrote: > I think we should not add blindly any language but should be open to add > couple of language like Scala. > In Bigdata world Scala/Python/Java are widely accepted. > > Thanks, > Milan Das > Interset > > On 2/13/18, 10:20 AM, "Weiss, Adam" wrote: > > I think it makes the most sense to me for us to publish a separate > repo with a module and nar build for now and post when it's available in > the users group. > > Thanks for the discussion everyone, hopefully we can start making som= e > helpful contributions soon. > > -Adam > > > On 2018/02/10 23:43:31, Tony Kurc gmail.com>> wrote: > > It is like Matt read my mind.> > > > > On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess > wrote:> > > > > > I'm fine with a vote, but I'll be voting to keep Java as the > single> > > > language for the (non-test) code. I share the same concerns as > many of> > > > the other folks as far as accepting other languages, it's mainly > the> > > > "slippery slope" argument that I don't want to turn into a> > > > JVM-language flame war. If Scala, why not Groovy? Certainly the> > > > syntax is closer to Java, and the community has accepted it as a > valid> > > > language for writing unit tests, although we stopped short for> > > > allowing it for the deployable NiFi codebase, for the same reason= s> > > > IIRC. If Scala and/or Groovy, why not Kotlin? The same argument= > > > > (albeit more tenuous) goes for Clojure and just about every other > JVM> > > > language (although I don't expect a call for LuaJ processors lol)= .> > > >> > > > Whether we decide to support various languages ad-hoc or not, I > would> > > > strenuously object to multiple/hybrid build systems for the > deployed> > > > artifacts. If I could switch NiFi completely to Gradle I would, > but I> > > > realize there are good reasons for not doing so (yet?) in the > Apache> > > > NiFi community, and I would never want any hybrid Maven/Gradle > build> > > > for the deployable code, likewise for SBT, Leiningen, etc. With a= > > > > custom Mojo for Maven NAR builds, and the complexity for hybrid > builds> > > > in general, I think this would create a maintenance nightmare.> > > >> > > > The language thing is a tough decision though, it's not awesome > that> > > > specifying a single language can be a barrier to a more diverse> > > > community, certainly Scala-based bundles would be more than > welcome in> > > > the overall NiFi ecosystem, I just think the cons outweigh the > pros> > > > for the baseline code. I've written Groovy processors/NARs using> > > > Gradle as the build system, and I'm good with keeping them in my > own> > > > repo, especially when the Extension Registry becomes a thing. I > can> > > > see the Extension Registry perhaps making this a moot point, but> > > > clearly we need to have this discussion in the meantime.> > > >> > > > Regards,> > > > Matt> > > >> > > >> > > > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande > wrote:> > > > > Wasn't there a warning trigger about the NiFi distro size from > Apache> > > > > recently? IMO, before talking alternative languages, solve the > modularity> > > > > and NAR distribution problem. I think the implementation of a > module> > > > won't> > > > > matter much then, the point being not everything has to go in > the core,> > > > > base distribution, but can still be easily sourced from a known > repo, for> > > > > example.> > > > >> > > > > I have a feeling NiFi 1.6+ can be approaching 2GB distro size > soon :)> > > > >> > > > > Andrew> > > > >> > > > > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee >> > > > wrote:> > > > >> > > > >> This probably necessitates a vote, yeah?> > > > >>> > > > >> Frankly, I=E2=80=99m usually happier writing Scala, and I=E2= =80=99ve not > encountered any> > > > >> problems using processors written in Scala, but I think it=E2= =80=99ll > be> > > > important> > > > >> to tread lightly.> > > > >>> > > > >> There=E2=80=99s a few things that pop into my head:> > > > >>> > > > >> - Maintainability and reviewability. A very very good Java > developer> > > > need> > > > >> not, by definition, either know how to write or identify good > Scala or> > > > spot> > > > >> problems and bugs.> > > > >> - Every Scala processor would either end up with a 5MB > scala-lang.jar> > > > >> packaged into the .nar or we=E2=80=99d have to start including= it in > the core> > > > >> somewhere, if it=E2=80=99s not. It=E2=80=99s possible it might= have already > gotten> > > > pulled> > > > >> up from other dependencies.> > > > >> - Style. There=E2=80=99s a tremendous amount of variation in S= cala > style because> > > > >> of its type system, implicits, macros, and functional nature. > There are> > > > >> very good people out there that can write good Scala that isn= =E2=80=99t> > > > readable by> > > > >> the 99%.> > > > >> - Binary compatibility. Scala tends to be a little more brazen > about> > > > >> breaking binary compatibility in major releases and those > happen a bit> > > > more> > > > >> often than with Java. That=E2=80=99s not a problem for any pot= ential > source> > > > code in> > > > >> the project, but it could present some dependency issues > someday.> > > > >> - Testing. There=E2=80=99s N > 1 test frameworks and testing s= tyles > within> > > > those,> > > > >> so there=E2=80=99s a lot of options for introducing more varia= bility > into the> > > > tests.> > > > >> - NiFi uses a lot of statics in setting up properties and > relationships> > > > >> and the like, and idiomatic Scala approaches that stuff a bit> > > > differently,> > > > >> so it=E2=80=99ll be necessary to impose some style guidelines = so there > isn=E2=80=99t too> > > > >> much variation.> > > > >>> > > > >> That said, there are some things that won=E2=80=99t be problem= atic:> > > > >>> > > > >> - As mentioned, processors written in Scala do just work.> > > > >> - The scala-maven-plugin works just fine allowing mixed > Java-Scala> > > > >> projects (btw, it=E2=80=99d probably be not super great to do = mixed > Java-Scala> > > > and> > > > >> mixed Maven-SBT though).> > > > >> - A lot of the above concerns could be addressed by having > clear style> > > > >> guidelines.> > > > >>> > > > >> Another thing: most of the projects I see deliver separate jar= s > for> > > > Scala> > > > >> components are delivering idiomatic APIs wrapping Java (or vic= e > versa).> > > > I> > > > >> think publishing a separate set of jars/nars for stuff > written in> > > > Scala> > > > >> would be odd since here it=E2=80=99d mostly be processors with= new > functionality> > > > >> and not functionality for using Scala. I could imagine a lib o= f> > > > implicits,> > > > >> traits, classes that could make the Scala development more > enjoyable.> > > > That> > > > >> probably would make sense to deliver that way.> > > > >>> > > > >> -joey> > > > >>> > > > >> On Feb 10, 2018, 10:33 AM -0600, Bryan Bende >, wrote:> > > > >> > I agree more with Andy about sticking with Java. The more > varying> > > > >> languages> > > > >> > used, the more challenging it is to maintain. Once the code > is part of> > > > >> the> > > > >> > Apache NiFi git repo, it is now the responsibility of the > committers> > > > and> > > > >> > PMC members to maintain it.> > > > >> >> > > > >> > I=E2=80=99d even say I am somewhat against the groovy/Spock = test code > that> > > > Andy> > > > >> > mentioned. I have frequently spent hours trying to fix a > Spock test> > > > that> > > > >> > broke from something I was working on. Every committer is > familiar> > > > with> > > > >> > JUnit, but only a couple know Spock. Just using this as an > example> > > > that> > > > >> > every committer knows Java, but only a couple probably know > Scala,> > > > >> Clojure,> > > > >> > etc.> > > > >> >> > > > >> > On Sat, Feb 10, 2018 at 10:25 AM Jeff > wrote:> > > > >> >> > > > >> > > +1 to Joe's response. If you can develop a component in > Groovy or> > > > Scala> > > > >> > > (or Clojure!) more quickly/comfortably, or if allowing > components> > > > >> written> > > > >> > > in other languages would encourage people to contribute > more, I'm> > > > all> > > > >> for> > > > >> > > it.> > > > >> > >> > > > >> > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt >> > > > wrote:> > > > >> > >> > > > >> > > > i personally would be ok with it for an > extension/processor> > > > provided> > > > >> it> > > > >> > > > integrates well with the build.> > > > >> > > >> > > > >> > > > i would agree with andys view for core framework stuff > but for> > > > >> > > extensions i> > > > >> > > > think we can do it like mikethomsen suggested.> > > > >> > > >> > > > >> > > > others?> > > > >> > > >> > > > >> > > > thanks> > > > >> > > > joe> > > > >> > > >> > > > >> > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" >> > > > >> wrote:> > > > >> > > >> > > > >> > > > > I'm just a community contributor, so take that FWIW, > but a> > > > >> compromise> > > > >> > > > might> > > > >> > > > > be to publish the Scala code as separate maven modules > to maven> > > > >> central> > > > >> > > > and> > > > >> > > > > then submit a thoroughly tested processor written in > Java. As> > > > long> > > > >> as> > > > >> > > you> > > > >> > > > > have enough unit and integration tests to give strong > coverage,> > > > I> > > > >> > > > wouldn't> > > > >> > > > > imagine anyone here would have issues reviewing it. If > the tests> > > > >> fail> > > > >> > > > > because of code issues in the external dependencies, > the obvious> > > > >> answer> > > > >> > > > is> > > > >> > > > > to just hold the PR until the tests pass.> > > > >> > > > >> > > > >> > > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <> > > > >> > > Adam.Weiss@perkinelmer.com com>> > > > >> > > > > wrote:> > > > >> > > > >> > > > >> > > > > > Devs,> > > > >> > > > > >> > > > >> > > > > > I have some interns starting with my team and we use > Scala> > > > >> internally> > > > >> > > > for> > > > >> > > > > > our work.> > > > >> > > > > > If I wanted to have them work to contribute some new= > > > > processors,> > > > >> > > would> > > > >> > > > > > they have to be Java to be included with the base> > > > distribution or> > > > >> > > could> > > > >> > > > > > they use Scala as long as it fit within your current > build and> > > > >> test> > > > >> > > > > process?> > > > >> > > > > >> > > > >> > > > > > Thanks,> > > > >> > > > > > -Adam> > > > >> > > > > >> > > > >> > > > >> > > > >> > > >> > > > >> > >> > > > >> > --> > > > >> > Sent from Gmail Mobile> > > > >>> > > >> > > > > Adam Weiss | Senior Manager, Digital Solutions R&D > PerkinElmer | Innovating for a Healthier World > adam.weiss@perkinelmer.com > Mobile: +1 716.429.7324 > 77 Goodell St. Suite 470, Buffalo, NY, 14203 > www.perkinelmer.com > > > > --001a113db248efa42405651b02ed--