flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Scala 2.10/2.11 Maven dependencies
Date Mon, 02 Nov 2015 19:52:13 GMT
+1 for Max' suggestion to fix that for 1.0 (next release).

Hot fixing of this thing so short before a release is a bit risky in my
opinion. It is easy to make errors (overlooking something, error not
visible because of cached older dependencies, ...), it happened more than
once during version upgrades, maven project re-organizations, etc.

Doing it after 0.10 and having a few week to let it sink in and errors
surface would probably be much safer...

On Mon, Nov 2, 2015 at 5:19 AM, Fabian Hueske <fhueske@gmail.com> wrote:

> Ah OK. Sorry, I misunderstood your intention.
>
> 2015-11-02 14:07 GMT+01:00 Maximilian Michels <mxm@apache.org>:
>
> > > That would mean to have "flink-java_2.10" and "flink-java_2.11"
> artifacts
> > > (and others that depend on flink-java and have no other Scala
> dependency)
> > > in the 0.10.0 release and only "flink-java" in the next 1.0 release.
> >
> >
> > My suggestion was to keep the Scala unsuffixed Scala 2.10 and add a
> > suffix for Scala 2.11. That's the way we are currently doing it (also
> > deployed on Maven like this). For the next release after 0.10, we can
> > do it properly.
> >
> > On Mon, Nov 2, 2015 at 1:30 PM, Chiwan Park <chiwanpark@apache.org>
> wrote:
> > > If we choose selective Scala version suffix for artifacts, we have to
> > tell which artifacts have the version suffix to newcomers. Some artifacts
> > such as "flink-java”, "flink-streaming-java" are easily recognized. But
> > IMO, knowing whether artifacts such as "flink-ml", "flink-clients",
> > "flink-table" have the version suffix or not is difficult for newcomers.
> > >
> > > This is why we are adding the version suffix to all Scala 2.11
> artifacts
> > currently. For Scala 2.10 artifacts, we aren’t adding the version suffix
> > for Flink with Java users.
> > >
> > > I’m for adding the version suffix to Scala 2.10 artifacts also. But I’m
> > not sure that removing the version suffix from Java-only artifacts would
> be
> > good. As I said above, It seems difficult for newcomers.
> > >
> > > Regards,
> > > Chiwan Park
> > >
> > > On November 2, 2015 at 8:19:15 PM, Fabian Hueske (fhueske@gmail.com)
> > wrote:
> > >
> > > That would mean to have "flink-java_2.10" and "flink-java_2.11"
> artifacts
> > > (and others that depend on flink-java and have no other Scala
> dependency)
> > > in the 0.10.0 release and only "flink-java" in the next 1.0 release.
> > >
> > > Do we want that?
> > >
> > > 2015-11-02 11:37 GMT+01:00 Maximilian Michels <mxm@apache.org>:
> > >
> > >> I'm for leaving it as-is and renaming all artifacts which depend on
> > >> Scala for the release following 0.10.
> > >>
> > >> On Mon, Nov 2, 2015 at 11:32 AM, Fabian Hueske <fhueske@gmail.com>
> > wrote:
> > >> > OK, let me try to summarize the discussion (and please correct me
> if I
> > >> got
> > >> > something wrong).
> > >> >
> > >> > 1) Flink deploys Scala 2.11 snapshot artifacts. Therefore, we have
> to
> > >> > release 2.11 artifacts for the 0.10.0 release version as well.
> > >> >
> > >> > 2) Everybody agrees to appropriately tag all artifacts that have a
> > >> > (transitive) Scala dependency. ATM, that would also include
> flink-java
> > >> > which is a bit awkward. The Scala dependency in flink-java
> originates
> > >> from
> > >> > the Chill library which is used to obtain a Kryo serializer which
is
> > >> > initialized with serializers for Scala classes. We could resolve
> this
> > >> issue
> > >> > by providing Java and Scala specific implementations of the Kryo
> > >> > serializers and have KryoTypeInfos for Java and Scala.
> > >> >
> > >> > The question to answer right now is, do we want to have "correctly"
> > >> labeled
> > >> > artifacts for the next 0.10.0 release or do we defer that for 1.0?
> > >> > If we want to solve it for 0.10.0 we need to cancel the current RC
> and
> > >> > provide a fix to remove the Scala dependency in flink-java, IMO.
> > >> >
> > >> > Opinions?
> > >> >
> > >> > Cheers, Fabian
> > >> >
> > >> > 2015-11-02 8:55 GMT+01:00 Stephan Ewen <sewen@apache.org>:
> > >> >
> > >> >> +1 for the approach discusses here, and for removing Scala
> > dependencies
> > >> >> from modules that can be Scala independent.
> > >> >>
> > >> >> It would be nice if pure Java users would not see any Scala
> > versioning
> > >> (on
> > >> >> flink-core, flink-java, later also flink-sreaming-java). I guess
> for
> > any
> > >> >> runtime-related parts (including flink-client and currently all
> > >> streaming
> > >> >> projects), we need the Scala versions...
> > >> >>
> > >> >> On Sun, Nov 1, 2015 at 9:29 AM, Maximilian Michels <mxm@apache.org
> >
> > >> wrote:
> > >> >>
> > >> >> > Good point. Didn't know that. We can still add them for the
> > release.
> > >> >> >
> > >> >> > On Sat, Oct 31, 2015 at 1:51 PM, Alexander Alexandrov
> > >> >> > <alexander.s.alexandrov@gmail.com> wrote:
> > >> >> > > My two cents - there are already Maven artifacts deployed
for
> > 2.11
> > >> in
> > >> >> the
> > >> >> > > SNAPSHOT repository. I think it might be confusing if
they
> > suddenly
> > >> >> > > disappear for the stable release.
> > >> >> > >
> > >> >> > >
> > >> >> > > 2015-10-29 11:58 GMT+01:00 Maximilian Michels <mxm@apache.org
> >:
> > >> >> > >
> > >> >> > >> Seems like we agree that we need artifacts for different
> > versions
> > >> of
> > >> >> > Scala
> > >> >> > >> on Maven. There also seems to be a preference for
including
> the
> > >> >> version
> > >> >> > in
> > >> >> > >> the artifact name.
> > >> >> > >>
> > >> >> > >> I've created an issue and marked it to be resolved
for 1.0.
> For
> > the
> > >> >> 0.10
> > >> >> > >> release, we will have binaries but no Maven artifacts.
The
> > biggest
> > >> >> > >> challenge I see is to remove Scala from as many
modules as
> > >> possible.
> > >> >> For
> > >> >> > >> example, flink-java depends on Scala at the moment..
> > >> >> > >>
> > >> >> > >> https://issues.apache.org/jira/browse/FLINK-2940
> > >> >> > >>
> > >> >> > >> On Wed, Oct 28, 2015 at 7:31 PM, Frederick F. Kautz
IV <
> > >> >> > fkautz@redhat.com>
> > >> >> > >> wrote:
> > >> >> > >>
> > >> >> > >> > No idea if I get a vote ;) Nevertheless, +1
to have binaries
> > for
> > >> >> both
> > >> >> > >> > versions in Maven and explicitly "scala versioned".
> > >> >> > >> >
> > >> >> > >> > Some background on this for those not as familiar
with scala
> > >> >> > versioning:
> > >> >> > >> >
> > >> >> > >> > It's considered best practice to label what
version of
> scala a
> > >> >> library
> > >> >> > >> > uses in the artifact id.
> > >> >> > >> >
> > >> >> > >> > The reason is compiled scala code is only compatible
with
> the
> > >> major
> > >> >> > >> > version of scala it was compiled for. For example,
a library
> > >> >> > compatible
> > >> >> > >> > with 2.10 is not compatible with 2.11. The
same will be true
> > with
> > >> >> 2.12
> > >> >> > >> once
> > >> >> > >> > it is released. Mixing versions will result
in undefined
> > behavior
> > >> >> > which
> > >> >> > >> > will likely manifest itself as runtime exceptions.
> > >> >> > >> >
> > >> >> > >> > The convention to fix this problem is for all
published
> > >> libraries to
> > >> >> > >> > specify the version of scala they are compatible
with.
> Leaving
> > >> out
> > >> >> the
> > >> >> > >> > scala version in a library is akin to saying
"We don't
> depend
> > on
> > >> >> scala
> > >> >> > >> for
> > >> >> > >> > this library, so feel free to use whatever
you want." Sbt
> > users
> > >> will
> > >> >> > >> > typically specify the version of scala they
use and tooling
> is
> > >> built
> > >> >> > >> around
> > >> >> > >> > ensuring consistency with the "%%" operator.
> > >> >> > >> >
> > >> >> > >> > E.g.
> > >> >> > >> >
> > >> >> > >> > scalaVersion := "2.11.4"
> > >> >> > >> >
> > >> >> > >> > // this resolves to to artifactID: "scalacheck_2.11"
> > >> >> > >> > libraryDependencies += "org.scalacheck" %%
"scalacheck" %
> > >> "1.12.0" %
> > >> >> > >> "test"
> > >> >> > >> >
> > >> >> > >> > The most important part of this is that the
scala version is
> > >> >> explicit
> > >> >> > >> > which eliminates the problem for downstream
users.
> > >> >> > >> >
> > >> >> > >> > Cheers,
> > >> >> > >> > Frederick
> > >> >> > >> >
> > >> >> > >> >
> > >> >> > >> > On 10/28/2015 10:55 AM, Fabian Hueske wrote:
> > >> >> > >> >
> > >> >> > >> >> +1 to have binaries for both versions in
Maven and as build
> > to
> > >> >> > download.
> > >> >> > >> >>
> > >> >> > >> >> 2015-10-26 17:11 GMT+01:00 Theodore Vasiloudis
<
> > >> >> > >> >> theodoros.vasiloudis@gmail.com>:
> > >> >> > >> >>
> > >> >> > >> >> +1 for having binaries, I'm working on
a Spark application
> > >> >> currently
> > >> >> > >> with
> > >> >> > >> >>> Scala 2.11 and having to rebuild everything
when deploying
> > >> e.g. to
> > >> >> > EC2
> > >> >> > >> >>> is a
> > >> >> > >> >>> pain.
> > >> >> > >> >>>
> > >> >> > >> >>> On Mon, Oct 26, 2015 at 4:22 PM, Ufuk
Celebi <
> > uce@apache.org>
> > >> >> > wrote:
> > >> >> > >> >>>
> > >> >> > >> >>> I agree with Till, but is this something
you want to
> > address in
> > >> >> this
> > >> >> > >> >>>> release already?
> > >> >> > >> >>>>
> > >> >> > >> >>>> I would postpone it to 1.0.0.
> > >> >> > >> >>>>
> > >> >> > >> >>>> – Ufuk
> > >> >> > >> >>>>
> > >> >> > >> >>>> On 26 Oct 2015, at 16:17, Till
Rohrmann <
> > trohrmann@apache.org
> > >> >
> > >> >> > wrote:
> > >> >> > >> >>>>>
> > >> >> > >> >>>>> I would be in favor of deploying
also Scala 2.11
> > artifacts to
> > >> >> > Maven
> > >> >> > >> >>>>>
> > >> >> > >> >>>> since
> > >> >> > >> >>>
> > >> >> > >> >>>> more and more people will try out
Flink with Scala 2.11.
> > >> Having
> > >> >> the
> > >> >> > >> >>>>> dependencies in the Maven repository
makes it
> considerably
> > >> >> easier
> > >> >> > for
> > >> >> > >> >>>>> people to get their Flink jobs
running.
> > >> >> > >> >>>>>
> > >> >> > >> >>>>> Furthermore, I observed that
people are not aware that
> our
> > >> >> > deployed
> > >> >> > >> >>>>> artifacts, e.g. flink-runtime,
are built with Scala
> 2.10.
> > As
> > >> a
> > >> >> > >> >>>>>
> > >> >> > >> >>>> consequence,
> > >> >> > >> >>>>
> > >> >> > >> >>>>> they mix flink dependencies
with other dependencies
> > pulling
> > >> in
> > >> >> > Scala
> > >> >> > >> >>>>>
> > >> >> > >> >>>> 2.11
> > >> >> > >> >>>
> > >> >> > >> >>>> and then they wonder that the program
crashes. It would
> be,
> > >> imho,
> > >> >> > >> >>>>>
> > >> >> > >> >>>> clearer
> > >> >> > >> >>>
> > >> >> > >> >>>> if all our dependencies which depend
on a specific Scala
> > >> version
> > >> >> > would
> > >> >> > >> >>>>>
> > >> >> > >> >>>> have
> > >> >> > >> >>>>
> > >> >> > >> >>>>> the corresponding Scala suffix
appended.
> > >> >> > >> >>>>>
> > >> >> > >> >>>>> Adding the 2.10 suffix would
also spare us the hassle of
> > >> >> upgrading
> > >> >> > >> to a
> > >> >> > >> >>>>> newer Scala version in the
future, because then the
> > artifacts
> > >> >> > >> wouldn't
> > >> >> > >> >>>>> share the same artifact name.
> > >> >> > >> >>>>>
> > >> >> > >> >>>>> Cheers,
> > >> >> > >> >>>>> Till
> > >> >> > >> >>>>>
> > >> >> > >> >>>>> On Mon, Oct 26, 2015 at 4:04
PM, Maximilian Michels <
> > >> >> > mxm@apache.org>
> > >> >> > >> >>>>>
> > >> >> > >> >>>> wrote:
> > >> >> > >> >>>>
> > >> >> > >> >>>>> Hi Flinksters,
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>>> We have recently committed
an easy way to change
> Flink's
> > >> Scala
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>> version.
> > >> >> > >> >>>
> > >> >> > >> >>>> The
> > >> >> > >> >>>>
> > >> >> > >> >>>>> question arises now whether
we should ship Scala 2.11 as
> > >> >> binaries
> > >> >> > and
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>> via
> > >> >> > >> >>>>
> > >> >> > >> >>>>> Maven. For the rc0, I created
all binaries twice, for
> > Scala
> > >> 2.10
> > >> >> > and
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>> 2.11.
> > >> >> > >> >>>>
> > >> >> > >> >>>>> However, I didn't create Maven
artifacts. This follows
> our
> > >> >> current
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>> shipping
> > >> >> > >> >>>>
> > >> >> > >> >>>>> strategy where we only ship
Hadoop1 and Hadoop 2.3.0
> Maven
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>> dependencies
> > >> >> > >> >>>
> > >> >> > >> >>>> but
> > >> >> > >> >>>>
> > >> >> > >> >>>>> additionally Hadoop 2.4, 2.6,
2.7 as binaries.
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>>> Should we also upload Maven
dependencies for Scala
> 2.11?
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>>> If so, the next question
arises: What version pattern
> > >> should we
> > >> >> > have
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>> for
> > >> >> > >> >>>
> > >> >> > >> >>>> the Flink Scala 2.11 dependencies?
For Hadoop, we append
> > >> -hadoop1
> > >> >> > to
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>> the
> > >> >> > >> >>>
> > >> >> > >> >>>> VERSION, e.g. artifactID=flink-core,
> version=0.9.1-hadoop1.
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>>> However, it is common practice
to append the suffix to
> > the
> > >> >> > >> artifactID
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>> of
> > >> >> > >> >>>
> > >> >> > >> >>>> the Maven dependency, e.g. artifactID=flink-core_2.11,
> > >> >> > version=0.9.1.
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>> This
> > >> >> > >> >>>>
> > >> >> > >> >>>>> has mostly historic reasons
but is widely used.
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>>> Whatever naming pattern
we choose, it should be
> > consistent.
> > >> I
> > >> >> > would
> > >> >> > >> be
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>> in
> > >> >> > >> >>>>
> > >> >> > >> >>>>> favor of changing our artifact
names to contain the
> Hadoop
> > >> and
> > >> >> > Scala
> > >> >> > >> >>>>>> version. This would also
imply that all Scala dependent
> > >> Maven
> > >> >> > >> modules
> > >> >> > >> >>>>>> receive a Scala suffix
(also the default Scala 2.10
> > >> modules).
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>>> Cheers,
> > >> >> > >> >>>>>> Max
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>>>
> > >> >> > >> >>>>
> > >> >> > >> >
> > >> >> > >>
> > >> >> >
> > >> >>
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message