beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré ...@nanthrax.net>
Subject Re: spark runner for Scala 2.10
Date Thu, 22 Feb 2018 17:45:36 GMT
OK, do you want me to provide a Scala 2.10 build for you ?

Regards
JB

On 02/22/2018 06:44 PM, Gary Dusbabek wrote:
> Jean-Baptiste,
> 
> Thanks for responding. I agree--it would be better to use Scala 2.11. I'm in the
> process of creating a Beam POC with an existing platform and upgrading
> everything in that platform to Scala 2.11 as a prerequisite is out of scope.
> 
> It would be helpful to know if Beam in it's current state is backward
> incompatible with Scala 2.10 for reasons other than the dependencies.
> 
> But if there is a way to make it work to enable a POC, I would appreciate some
> pointers, as it doesn't seem to be as simple as changing the "*_2.11" references
> in the poms.
> 
> Cheers,
> 
> Gary.
> 
> 
> 
> On Thu, Feb 22, 2018 at 5:34 PM, Jean-Baptiste Onofré <jb@nanthrax.net
> <mailto:jb@nanthrax.net>> wrote:
> 
>     Hi Gary,
> 
>     Beam 2.3.0 and the Spark runner use Scala 2.11.
> 
>     I can help you to have a smooth transition by creating a local branch using
>     Scala 2.10. However,  I strongly advice to upgrade to 2.11 as some other part of
>     Beam  (other runners and IOs) use 2.11 already.
> 
>     Regards
>     JB
> 
>     On 02/22/2018 05:55 PM, Gary Dusbabek wrote:
>     > Hi,
>     >
>     > My apologies if this belongs on the dev list. If it does, let me know and I'll
>     > shoot things over that way...
>     >
>     > For the last day or so, I've been trying to create a Spark Runner that
>     will work
>     > on older deployments using Scala 2.10. I've taken a few approaches:
>     >
>     > 1. selectively changing a few dependencies in beam-runners-spark.pom (and
>     a few
>     > other places in the parent)
>     > 2. updating every dependency that references *_2.11 to be *_2.10
>     >
>     > In the former case the sticking point in both cases is that there is a library
>     > incompatibility with jackson-module-scala_2.xx. In the latter case there is
a
>     > problem with SourceRDD.SourcePartitioning not [correctly] implementing
>     > `equals(...)` from the parent trait.
>     >
>     > Posts on the mailing list made me think that the move to Scala 2.11
>     started only
>     > last fall, so I figured it should be easy to make the switch back.
>     >
>     > However, I have a feeling that it could be the case that I just don't
>     understand
>     > the Beam build system well enough to produce the right outcome (a custom
>     version
>     > that can be used with older Scala).
>     >
>     > Is there a correct or better way of achieving this?
>     >
>     > Kind Regards,
>     >
>     > Gary Dusbabek
> 
>     --
>     Jean-Baptiste Onofré
>     jbonofre@apache.org <mailto:jbonofre@apache.org>
>     http://blog.nanthrax.net
>     Talend - http://www.talend.com
> 
> 

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Mime
View raw message