flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: Scala 2.10/2.11 Maven dependencies
Date Mon, 02 Nov 2015 10:37:28 GMT
I'm for leaving it as-is and renaming all artifacts which depend on
Scala for the release following 0.10.

On Mon, Nov 2, 2015 at 11:32 AM, Fabian Hueske <fhueske@gmail.com> wrote:
> OK, let me try to summarize the discussion (and please correct me if I got
> something wrong).
>
> 1) Flink deploys Scala 2.11 snapshot artifacts. Therefore, we have to
> release 2.11 artifacts for the 0.10.0 release version as well.
>
> 2) Everybody agrees to appropriately tag all artifacts that have a
> (transitive) Scala dependency. ATM, that would also include flink-java
> which is a bit awkward. The Scala dependency in flink-java originates from
> the Chill library which is used to obtain a Kryo serializer which is
> initialized with serializers for Scala classes. We could resolve this issue
> by providing Java and Scala specific implementations of the Kryo
> serializers and have KryoTypeInfos for Java and Scala.
>
> The question to answer right now is, do we want to have "correctly" labeled
> artifacts for the next 0.10.0 release or do we defer that for 1.0?
> If we want to solve it for 0.10.0 we need to cancel the current RC and
> provide a fix to remove the Scala dependency in flink-java, IMO.
>
> Opinions?
>
> Cheers, Fabian
>
> 2015-11-02 8:55 GMT+01:00 Stephan Ewen <sewen@apache.org>:
>
>> +1 for the approach discusses here, and for removing Scala dependencies
>> from modules that can be Scala independent.
>>
>> It would be nice if pure Java users would not see any Scala versioning (on
>> flink-core, flink-java, later also flink-sreaming-java). I guess for any
>> runtime-related parts (including flink-client and currently all streaming
>> projects), we need the Scala versions...
>>
>> On Sun, Nov 1, 2015 at 9:29 AM, Maximilian Michels <mxm@apache.org> wrote:
>>
>> > Good point. Didn't know that. We can still add them for the release.
>> >
>> > On Sat, Oct 31, 2015 at 1:51 PM, Alexander Alexandrov
>> > <alexander.s.alexandrov@gmail.com> wrote:
>> > > My two cents - there are already Maven artifacts deployed for 2.11 in
>> the
>> > > SNAPSHOT repository. I think it might be confusing if they suddenly
>> > > disappear for the stable release.
>> > >
>> > >
>> > > 2015-10-29 11:58 GMT+01:00 Maximilian Michels <mxm@apache.org>:
>> > >
>> > >> Seems like we agree that we need artifacts for different versions of
>> > Scala
>> > >> on Maven. There also seems to be a preference for including the
>> version
>> > in
>> > >> the artifact name.
>> > >>
>> > >> I've created an issue and marked it to be resolved for 1.0. For the
>> 0.10
>> > >> release, we will have binaries but no Maven artifacts. The biggest
>> > >> challenge I see is to remove Scala from as many modules as possible.
>> For
>> > >> example, flink-java depends on Scala at the moment..
>> > >>
>> > >> https://issues.apache.org/jira/browse/FLINK-2940
>> > >>
>> > >> On Wed, Oct 28, 2015 at 7:31 PM, Frederick F. Kautz IV <
>> > fkautz@redhat.com>
>> > >> wrote:
>> > >>
>> > >> > No idea if I get a vote ;) Nevertheless, +1 to have binaries for
>> both
>> > >> > versions in Maven and explicitly "scala versioned".
>> > >> >
>> > >> > Some background on this for those not as familiar with scala
>> > versioning:
>> > >> >
>> > >> > It's considered best practice to label what version of scala a
>> library
>> > >> > uses in the artifact id.
>> > >> >
>> > >> > The reason is compiled scala code is only compatible with the
major
>> > >> > version of scala it was compiled for. For example, a library
>> > compatible
>> > >> > with 2.10 is not compatible with 2.11. The same will be true with
>> 2.12
>> > >> once
>> > >> > it is released. Mixing versions will result in undefined behavior
>> > which
>> > >> > will likely manifest itself as runtime exceptions.
>> > >> >
>> > >> > The convention to fix this problem is for all published libraries
to
>> > >> > specify the version of scala they are compatible with. Leaving
out
>> the
>> > >> > scala version in a library is akin to saying "We don't depend
on
>> scala
>> > >> for
>> > >> > this library, so feel free to use whatever you want." Sbt users
will
>> > >> > typically specify the version of scala they use and tooling is
built
>> > >> around
>> > >> > ensuring consistency with the "%%" operator.
>> > >> >
>> > >> > E.g.
>> > >> >
>> > >> > scalaVersion := "2.11.4"
>> > >> >
>> > >> > // this resolves to to artifactID: "scalacheck_2.11"
>> > >> > libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.12.0"
%
>> > >> "test"
>> > >> >
>> > >> > The most important part of this is that the scala version is
>> explicit
>> > >> > which eliminates the problem for downstream users.
>> > >> >
>> > >> > Cheers,
>> > >> > Frederick
>> > >> >
>> > >> >
>> > >> > On 10/28/2015 10:55 AM, Fabian Hueske wrote:
>> > >> >
>> > >> >> +1 to have binaries for both versions in Maven and as build
to
>> > download.
>> > >> >>
>> > >> >> 2015-10-26 17:11 GMT+01:00 Theodore Vasiloudis <
>> > >> >> theodoros.vasiloudis@gmail.com>:
>> > >> >>
>> > >> >> +1 for having binaries, I'm working on a Spark application
>> currently
>> > >> with
>> > >> >>> Scala 2.11 and having to rebuild everything when deploying
e.g. to
>> > EC2
>> > >> >>> is a
>> > >> >>> pain.
>> > >> >>>
>> > >> >>> On Mon, Oct 26, 2015 at 4:22 PM, Ufuk Celebi <uce@apache.org>
>> > wrote:
>> > >> >>>
>> > >> >>> I agree with Till, but is this something you want to address
in
>> this
>> > >> >>>> release already?
>> > >> >>>>
>> > >> >>>> I would postpone it to 1.0.0.
>> > >> >>>>
>> > >> >>>> – Ufuk
>> > >> >>>>
>> > >> >>>> On 26 Oct 2015, at 16:17, Till Rohrmann <trohrmann@apache.org>
>> > wrote:
>> > >> >>>>>
>> > >> >>>>> I would be in favor of deploying also Scala 2.11
artifacts to
>> > Maven
>> > >> >>>>>
>> > >> >>>> since
>> > >> >>>
>> > >> >>>> more and more people will try out Flink with Scala
2.11. Having
>> the
>> > >> >>>>> dependencies in the Maven repository makes it
considerably
>> easier
>> > for
>> > >> >>>>> people to get their Flink jobs running.
>> > >> >>>>>
>> > >> >>>>> Furthermore, I observed that people are not aware
that our
>> > deployed
>> > >> >>>>> artifacts, e.g. flink-runtime, are built with
Scala 2.10. As a
>> > >> >>>>>
>> > >> >>>> consequence,
>> > >> >>>>
>> > >> >>>>> they mix flink dependencies with other dependencies
pulling in
>> > Scala
>> > >> >>>>>
>> > >> >>>> 2.11
>> > >> >>>
>> > >> >>>> and then they wonder that the program crashes. It
would be, imho,
>> > >> >>>>>
>> > >> >>>> clearer
>> > >> >>>
>> > >> >>>> if all our dependencies which depend on a specific
Scala version
>> > would
>> > >> >>>>>
>> > >> >>>> have
>> > >> >>>>
>> > >> >>>>> the corresponding Scala suffix appended.
>> > >> >>>>>
>> > >> >>>>> Adding the 2.10 suffix would also spare us the
hassle of
>> upgrading
>> > >> to a
>> > >> >>>>> newer Scala version in the future, because then
the artifacts
>> > >> wouldn't
>> > >> >>>>> share the same artifact name.
>> > >> >>>>>
>> > >> >>>>> Cheers,
>> > >> >>>>> Till
>> > >> >>>>>
>> > >> >>>>> On Mon, Oct 26, 2015 at 4:04 PM, Maximilian Michels
<
>> > mxm@apache.org>
>> > >> >>>>>
>> > >> >>>> wrote:
>> > >> >>>>
>> > >> >>>>> Hi Flinksters,
>> > >> >>>>>>
>> > >> >>>>>> We have recently committed an easy way to
change Flink's Scala
>> > >> >>>>>>
>> > >> >>>>> version.
>> > >> >>>
>> > >> >>>> The
>> > >> >>>>
>> > >> >>>>> question arises now whether we should ship Scala
2.11 as
>> binaries
>> > and
>> > >> >>>>>>
>> > >> >>>>> via
>> > >> >>>>
>> > >> >>>>> Maven. For the rc0, I created all binaries twice,
for Scala 2.10
>> > and
>> > >> >>>>>>
>> > >> >>>>> 2.11.
>> > >> >>>>
>> > >> >>>>> However, I didn't create Maven artifacts. This
follows our
>> current
>> > >> >>>>>>
>> > >> >>>>> shipping
>> > >> >>>>
>> > >> >>>>> strategy where we only ship Hadoop1 and Hadoop
2.3.0 Maven
>> > >> >>>>>>
>> > >> >>>>> dependencies
>> > >> >>>
>> > >> >>>> but
>> > >> >>>>
>> > >> >>>>> additionally Hadoop 2.4, 2.6, 2.7 as binaries.
>> > >> >>>>>>
>> > >> >>>>>> Should we also upload Maven dependencies for
Scala 2.11?
>> > >> >>>>>>
>> > >> >>>>>> If so, the next question arises: What version
pattern should we
>> > have
>> > >> >>>>>>
>> > >> >>>>> for
>> > >> >>>
>> > >> >>>> the Flink Scala 2.11 dependencies? For Hadoop, we
append -hadoop1
>> > to
>> > >> >>>>>>
>> > >> >>>>> the
>> > >> >>>
>> > >> >>>> VERSION, e.g. artifactID=flink-core, version=0.9.1-hadoop1.
>> > >> >>>>>>
>> > >> >>>>>> However, it is common practice to append the
suffix to the
>> > >> artifactID
>> > >> >>>>>>
>> > >> >>>>> of
>> > >> >>>
>> > >> >>>> the Maven dependency, e.g. artifactID=flink-core_2.11,
>> > version=0.9.1.
>> > >> >>>>>>
>> > >> >>>>> This
>> > >> >>>>
>> > >> >>>>> has mostly historic reasons but is widely used.
>> > >> >>>>>>
>> > >> >>>>>> Whatever naming pattern we choose, it should
be consistent. I
>> > would
>> > >> be
>> > >> >>>>>>
>> > >> >>>>> in
>> > >> >>>>
>> > >> >>>>> favor of changing our artifact names to contain
the Hadoop and
>> > Scala
>> > >> >>>>>> version. This would also imply that all Scala
dependent Maven
>> > >> modules
>> > >> >>>>>> receive a Scala suffix (also the default Scala
2.10 modules).
>> > >> >>>>>>
>> > >> >>>>>> Cheers,
>> > >> >>>>>> Max
>> > >> >>>>>>
>> > >> >>>>>>
>> > >> >>>>
>> > >> >
>> > >>
>> >
>>

Mime
View raw message