Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAGco--YikVpNdg+ppa9e1o8jUAS1Je1okVbtAq6g0pLy0JcXwg@mail.gmail.com>
References: 
 <CAGco--a94WW__bxUGAco+R6zyuj1iTGP2UUoV-=XLgpAhQXaMA@mail.gmail.com>
 <CAC27z=MNyQQFoG2zrvJhzP95XL-z_kkn1k4qwUm8O2i_B_00Rg@mail.gmail.com>
 <D83DE763-DAA8-4111-A174-7AD6319A525D@apache.org>
 <CAGLgxbYfpmibfckaSLZNZSKX7vHVkGqJTsEGAKTEzViY4qpPyA@mail.gmail.com>
 <CAAdrtT1wVAwVTyFfP=b415Gxt6MLTPqJh+OgxadwN2J4wAgoiA@mail.gmail.com>
 <56311476.8080503@redhat.com>
 <CAGco--aNPrGTJD+nSK0kRQN4qLYBwbEz0VdKpZSf5e5v5pJYeA@mail.gmail.com>
 <CAEXqXcZgzD8Sr2AYwyM9QRpV+wmrLT4i_Osxxqua4uW1QmTYwA@mail.gmail.com>
 <CAGco--bW4aQ4UPq2=JkLcdd+-gDxvKW3gx+0s=URWJ5zAV3KFw@mail.gmail.com>
 <CANC1h_t9hjPMbzcFZgeM_qt_OZ=8vvnsaarF31CGfy+ORBoLVg@mail.gmail.com>
 <CAAdrtT24a97ULGgxDrjCMaHY=nyXKyKQj+z83S+POc726gKGOw@mail.gmail.com>
 <CAGco--YikVpNdg+ppa9e1o8jUAS1Je1okVbtAq6g0pLy0JcXwg@mail.gmail.com>
From: Fabian Hueske <fhueske@gmail.com>
Date: Mon, 2 Nov 2015 12:18:38 +0100
Message-ID: 
 <CAAdrtT0Jcbm_Sc1+yMrH6mqpy6MeA1Bzb24dK14dPZGqY4MXFA@mail.gmail.com>
Subject: Re: Scala 2.10/2.11 Maven dependencies
To: "dev@flink.apache.org" <dev@flink.apache.org>
Content-Type: multipart/alternative; boundary=001a113dafb21b62d305238cf22e

--001a113dafb21b62d305238cf22e
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

That would mean to have "flink-java_2.10" and "flink-java_2.11" artifacts
(and others that depend on flink-java and have no other Scala dependency)
in the 0.10.0 release and only "flink-java" in the next 1.0 release.

Do we want that?

2015-11-02 11:37 GMT+01:00 Maximilian Michels <mxm@apache.org>:

> I'm for leaving it as-is and renaming all artifacts which depend on
> Scala for the release following 0.10.
>
> On Mon, Nov 2, 2015 at 11:32 AM, Fabian Hueske <fhueske@gmail.com> wrote:
> > OK, let me try to summarize the discussion (and please correct me if I
> got
> > something wrong).
> >
> > 1) Flink deploys Scala 2.11 snapshot artifacts. Therefore, we have to
> > release 2.11 artifacts for the 0.10.0 release version as well.
> >
> > 2) Everybody agrees to appropriately tag all artifacts that have a
> > (transitive) Scala dependency. ATM, that would also include flink-java
> > which is a bit awkward. The Scala dependency in flink-java originates
> from
> > the Chill library which is used to obtain a Kryo serializer which is
> > initialized with serializers for Scala classes. We could resolve this
> issue
> > by providing Java and Scala specific implementations of the Kryo
> > serializers and have KryoTypeInfos for Java and Scala.
> >
> > The question to answer right now is, do we want to have "correctly"
> labeled
> > artifacts for the next 0.10.0 release or do we defer that for 1.0?
> > If we want to solve it for 0.10.0 we need to cancel the current RC and
> > provide a fix to remove the Scala dependency in flink-java, IMO.
> >
> > Opinions?
> >
> > Cheers, Fabian
> >
> > 2015-11-02 8:55 GMT+01:00 Stephan Ewen <sewen@apache.org>:
> >
> >> +1 for the approach discusses here, and for removing Scala dependencie=
s
> >> from modules that can be Scala independent.
> >>
> >> It would be nice if pure Java users would not see any Scala versioning
> (on
> >> flink-core, flink-java, later also flink-sreaming-java). I guess for a=
ny
> >> runtime-related parts (including flink-client and currently all
> streaming
> >> projects), we need the Scala versions...
> >>
> >> On Sun, Nov 1, 2015 at 9:29 AM, Maximilian Michels <mxm@apache.org>
> wrote:
> >>
> >> > Good point. Didn't know that. We can still add them for the release.
> >> >
> >> > On Sat, Oct 31, 2015 at 1:51 PM, Alexander Alexandrov
> >> > <alexander.s.alexandrov@gmail.com> wrote:
> >> > > My two cents - there are already Maven artifacts deployed for 2.11
> in
> >> the
> >> > > SNAPSHOT repository. I think it might be confusing if they suddenl=
y
> >> > > disappear for the stable release.
> >> > >
> >> > >
> >> > > 2015-10-29 11:58 GMT+01:00 Maximilian Michels <mxm@apache.org>:
> >> > >
> >> > >> Seems like we agree that we need artifacts for different versions
> of
> >> > Scala
> >> > >> on Maven. There also seems to be a preference for including the
> >> version
> >> > in
> >> > >> the artifact name.
> >> > >>
> >> > >> I've created an issue and marked it to be resolved for 1.0. For t=
he
> >> 0.10
> >> > >> release, we will have binaries but no Maven artifacts. The bigges=
t
> >> > >> challenge I see is to remove Scala from as many modules as
> possible.
> >> For
> >> > >> example, flink-java depends on Scala at the moment..
> >> > >>
> >> > >> https://issues.apache.org/jira/browse/FLINK-2940
> >> > >>
> >> > >> On Wed, Oct 28, 2015 at 7:31 PM, Frederick F. Kautz IV <
> >> > fkautz@redhat.com>
> >> > >> wrote:
> >> > >>
> >> > >> > No idea if I get a vote ;) Nevertheless, +1 to have binaries fo=
r
> >> both
> >> > >> > versions in Maven and explicitly "scala versioned".
> >> > >> >
> >> > >> > Some background on this for those not as familiar with scala
> >> > versioning:
> >> > >> >
> >> > >> > It's considered best practice to label what version of scala a
> >> library
> >> > >> > uses in the artifact id.
> >> > >> >
> >> > >> > The reason is compiled scala code is only compatible with the
> major
> >> > >> > version of scala it was compiled for. For example, a library
> >> > compatible
> >> > >> > with 2.10 is not compatible with 2.11. The same will be true wi=
th
> >> 2.12
> >> > >> once
> >> > >> > it is released. Mixing versions will result in undefined behavi=
or
> >> > which
> >> > >> > will likely manifest itself as runtime exceptions.
> >> > >> >
> >> > >> > The convention to fix this problem is for all published
> libraries to
> >> > >> > specify the version of scala they are compatible with. Leaving
> out
> >> the
> >> > >> > scala version in a library is akin to saying "We don't depend o=
n
> >> scala
> >> > >> for
> >> > >> > this library, so feel free to use whatever you want." Sbt users
> will
> >> > >> > typically specify the version of scala they use and tooling is
> built
> >> > >> around
> >> > >> > ensuring consistency with the "%%" operator.
> >> > >> >
> >> > >> > E.g.
> >> > >> >
> >> > >> > scalaVersion :=3D "2.11.4"
> >> > >> >
> >> > >> > // this resolves to to artifactID: "scalacheck_2.11"
> >> > >> > libraryDependencies +=3D "org.scalacheck" %% "scalacheck" %
> "1.12.0" %
> >> > >> "test"
> >> > >> >
> >> > >> > The most important part of this is that the scala version is
> >> explicit
> >> > >> > which eliminates the problem for downstream users.
> >> > >> >
> >> > >> > Cheers,
> >> > >> > Frederick
> >> > >> >
> >> > >> >
> >> > >> > On 10/28/2015 10:55 AM, Fabian Hueske wrote:
> >> > >> >
> >> > >> >> +1 to have binaries for both versions in Maven and as build to
> >> > download.
> >> > >> >>
> >> > >> >> 2015-10-26 17:11 GMT+01:00 Theodore Vasiloudis <
> >> > >> >> theodoros.vasiloudis@gmail.com>:
> >> > >> >>
> >> > >> >> +1 for having binaries, I'm working on a Spark application
> >> currently
> >> > >> with
> >> > >> >>> Scala 2.11 and having to rebuild everything when deploying
> e.g. to
> >> > EC2
> >> > >> >>> is a
> >> > >> >>> pain.
> >> > >> >>>
> >> > >> >>> On Mon, Oct 26, 2015 at 4:22 PM, Ufuk Celebi <uce@apache.org>
> >> > wrote:
> >> > >> >>>
> >> > >> >>> I agree with Till, but is this something you want to address =
in
> >> this
> >> > >> >>>> release already?
> >> > >> >>>>
> >> > >> >>>> I would postpone it to 1.0.0.
> >> > >> >>>>
> >> > >> >>>> =E2=80=93 Ufuk
> >> > >> >>>>
> >> > >> >>>> On 26 Oct 2015, at 16:17, Till Rohrmann <trohrmann@apache.or=
g
> >
> >> > wrote:
> >> > >> >>>>>
> >> > >> >>>>> I would be in favor of deploying also Scala 2.11 artifacts =
to
> >> > Maven
> >> > >> >>>>>
> >> > >> >>>> since
> >> > >> >>>
> >> > >> >>>> more and more people will try out Flink with Scala 2.11.
> Having
> >> the
> >> > >> >>>>> dependencies in the Maven repository makes it considerably
> >> easier
> >> > for
> >> > >> >>>>> people to get their Flink jobs running.
> >> > >> >>>>>
> >> > >> >>>>> Furthermore, I observed that people are not aware that our
> >> > deployed
> >> > >> >>>>> artifacts, e.g. flink-runtime, are built with Scala 2.10. A=
s
> a
> >> > >> >>>>>
> >> > >> >>>> consequence,
> >> > >> >>>>
> >> > >> >>>>> they mix flink dependencies with other dependencies pulling
> in
> >> > Scala
> >> > >> >>>>>
> >> > >> >>>> 2.11
> >> > >> >>>
> >> > >> >>>> and then they wonder that the program crashes. It would be,
> imho,
> >> > >> >>>>>
> >> > >> >>>> clearer
> >> > >> >>>
> >> > >> >>>> if all our dependencies which depend on a specific Scala
> version
> >> > would
> >> > >> >>>>>
> >> > >> >>>> have
> >> > >> >>>>
> >> > >> >>>>> the corresponding Scala suffix appended.
> >> > >> >>>>>
> >> > >> >>>>> Adding the 2.10 suffix would also spare us the hassle of
> >> upgrading
> >> > >> to a
> >> > >> >>>>> newer Scala version in the future, because then the artifac=
ts
> >> > >> wouldn't
> >> > >> >>>>> share the same artifact name.
> >> > >> >>>>>
> >> > >> >>>>> Cheers,
> >> > >> >>>>> Till
> >> > >> >>>>>
> >> > >> >>>>> On Mon, Oct 26, 2015 at 4:04 PM, Maximilian Michels <
> >> > mxm@apache.org>
> >> > >> >>>>>
> >> > >> >>>> wrote:
> >> > >> >>>>
> >> > >> >>>>> Hi Flinksters,
> >> > >> >>>>>>
> >> > >> >>>>>> We have recently committed an easy way to change Flink's
> Scala
> >> > >> >>>>>>
> >> > >> >>>>> version.
> >> > >> >>>
> >> > >> >>>> The
> >> > >> >>>>
> >> > >> >>>>> question arises now whether we should ship Scala 2.11 as
> >> binaries
> >> > and
> >> > >> >>>>>>
> >> > >> >>>>> via
> >> > >> >>>>
> >> > >> >>>>> Maven. For the rc0, I created all binaries twice, for Scala
> 2.10
> >> > and
> >> > >> >>>>>>
> >> > >> >>>>> 2.11.
> >> > >> >>>>
> >> > >> >>>>> However, I didn't create Maven artifacts. This follows our
> >> current
> >> > >> >>>>>>
> >> > >> >>>>> shipping
> >> > >> >>>>
> >> > >> >>>>> strategy where we only ship Hadoop1 and Hadoop 2.3.0 Maven
> >> > >> >>>>>>
> >> > >> >>>>> dependencies
> >> > >> >>>
> >> > >> >>>> but
> >> > >> >>>>
> >> > >> >>>>> additionally Hadoop 2.4, 2.6, 2.7 as binaries.
> >> > >> >>>>>>
> >> > >> >>>>>> Should we also upload Maven dependencies for Scala 2.11?
> >> > >> >>>>>>
> >> > >> >>>>>> If so, the next question arises: What version pattern
> should we
> >> > have
> >> > >> >>>>>>
> >> > >> >>>>> for
> >> > >> >>>
> >> > >> >>>> the Flink Scala 2.11 dependencies? For Hadoop, we append
> -hadoop1
> >> > to
> >> > >> >>>>>>
> >> > >> >>>>> the
> >> > >> >>>
> >> > >> >>>> VERSION, e.g. artifactID=3Dflink-core, version=3D0.9.1-hadoo=
p1.
> >> > >> >>>>>>
> >> > >> >>>>>> However, it is common practice to append the suffix to the
> >> > >> artifactID
> >> > >> >>>>>>
> >> > >> >>>>> of
> >> > >> >>>
> >> > >> >>>> the Maven dependency, e.g. artifactID=3Dflink-core_2.11,
> >> > version=3D0.9.1.
> >> > >> >>>>>>
> >> > >> >>>>> This
> >> > >> >>>>
> >> > >> >>>>> has mostly historic reasons but is widely used.
> >> > >> >>>>>>
> >> > >> >>>>>> Whatever naming pattern we choose, it should be consistent=
.
> I
> >> > would
> >> > >> be
> >> > >> >>>>>>
> >> > >> >>>>> in
> >> > >> >>>>
> >> > >> >>>>> favor of changing our artifact names to contain the Hadoop
> and
> >> > Scala
> >> > >> >>>>>> version. This would also imply that all Scala dependent
> Maven
> >> > >> modules
> >> > >> >>>>>> receive a Scala suffix (also the default Scala 2.10
> modules).
> >> > >> >>>>>>
> >> > >> >>>>>> Cheers,
> >> > >> >>>>>> Max
> >> > >> >>>>>>
> >> > >> >>>>>>
> >> > >> >>>>
> >> > >> >
> >> > >>
> >> >
> >>
>

--001a113dafb21b62d305238cf22e--