From user-return-2995-archive-asf-public=cust-asf.ponee.io@beam.apache.org Thu Feb 22 18:45:47 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id F390C18064E for ; Thu, 22 Feb 2018 18:45:46 +0100 (CET) Received: (qmail 26907 invoked by uid 500); 22 Feb 2018 17:45:45 -0000 Mailing-List: contact user-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@beam.apache.org Delivered-To: mailing list user@beam.apache.org Received: (qmail 26859 invoked by uid 99); 22 Feb 2018 17:45:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Feb 2018 17:45:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 5235E1A0389 for ; Thu, 22 Feb 2018 17:45:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.78 X-Spam-Level: X-Spam-Status: No, score=0.78 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 3JyBmlMyHtxq for ; Thu, 22 Feb 2018 17:45:43 +0000 (UTC) Received: from relay2-d.mail.gandi.net (relay2-d.mail.gandi.net [217.70.183.194]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id D77DB5F522 for ; Thu, 22 Feb 2018 17:45:42 +0000 (UTC) X-Originating-IP: 109.122.146.26 Received: from [192.168.1.39] (26.146.122.109.rev.sfr.net [109.122.146.26]) (Authenticated sender: jb@nanthrax.net) by relay2-d.mail.gandi.net (Postfix) with ESMTPSA id 2EDDDC5A73 for ; Thu, 22 Feb 2018 18:45:40 +0100 (CET) Subject: Re: spark runner for Scala 2.10 To: user@beam.apache.org References: From: =?UTF-8?Q?Jean-Baptiste_Onofr=c3=a9?= Message-ID: <10ab6cd0-a62c-9c2f-959c-29caa6d6624f@nanthrax.net> Date: Thu, 22 Feb 2018 18:45:36 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit OK, do you want me to provide a Scala 2.10 build for you ? Regards JB On 02/22/2018 06:44 PM, Gary Dusbabek wrote: > Jean-Baptiste, > > Thanks for responding. I agree--it would be better to use Scala 2.11. I'm in the > process of creating a Beam POC with an existing platform and upgrading > everything in that platform to Scala 2.11 as a prerequisite is out of scope. > > It would be helpful to know if Beam in it's current state is backward > incompatible with Scala 2.10 for reasons other than the dependencies. > > But if there is a way to make it work to enable a POC, I would appreciate some > pointers, as it doesn't seem to be as simple as changing the "*_2.11" references > in the poms. > > Cheers, > > Gary. > > > > On Thu, Feb 22, 2018 at 5:34 PM, Jean-Baptiste Onofré > wrote: > > Hi Gary, > > Beam 2.3.0 and the Spark runner use Scala 2.11. > > I can help you to have a smooth transition by creating a local branch using > Scala 2.10. However,  I strongly advice to upgrade to 2.11 as some other part of > Beam  (other runners and IOs) use 2.11 already. > > Regards > JB > > On 02/22/2018 05:55 PM, Gary Dusbabek wrote: > > Hi, > > > > My apologies if this belongs on the dev list. If it does, let me know and I'll > > shoot things over that way... > > > > For the last day or so, I've been trying to create a Spark Runner that > will work > > on older deployments using Scala 2.10. I've taken a few approaches: > > > > 1. selectively changing a few dependencies in beam-runners-spark.pom (and > a few > > other places in the parent) > > 2. updating every dependency that references *_2.11 to be *_2.10 > > > > In the former case the sticking point in both cases is that there is a library > > incompatibility with jackson-module-scala_2.xx. In the latter case there is a > > problem with SourceRDD.SourcePartitioning not [correctly] implementing > > `equals(...)` from the parent trait. > > > > Posts on the mailing list made me think that the move to Scala 2.11 > started only > > last fall, so I figured it should be easy to make the switch back. > > > > However, I have a feeling that it could be the case that I just don't > understand > > the Beam build system well enough to produce the right outcome (a custom > version > > that can be used with older Scala). > > > > Is there a correct or better way of achieving this? > > > > Kind Regards, > > > > Gary Dusbabek > > -- > Jean-Baptiste Onofré > jbonofre@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com > > -- Jean-Baptiste Onofré jbonofre@apache.org http://blog.nanthrax.net Talend - http://www.talend.com