flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Scala 2.10/2.11 Maven dependencies
Date Mon, 02 Nov 2015 13:19:07 GMT
Ah OK. Sorry, I misunderstood your intention.

2015-11-02 14:07 GMT+01:00 Maximilian Michels <mxm@apache.org>:

> > That would mean to have "flink-java_2.10" and "flink-java_2.11" artifacts
> > (and others that depend on flink-java and have no other Scala dependency)
> > in the 0.10.0 release and only "flink-java" in the next 1.0 release.
>
>
> My suggestion was to keep the Scala unsuffixed Scala 2.10 and add a
> suffix for Scala 2.11. That's the way we are currently doing it (also
> deployed on Maven like this). For the next release after 0.10, we can
> do it properly.
>
> On Mon, Nov 2, 2015 at 1:30 PM, Chiwan Park <chiwanpark@apache.org> wrote:
> > If we choose selective Scala version suffix for artifacts, we have to
> tell which artifacts have the version suffix to newcomers. Some artifacts
> such as "flink-java”, "flink-streaming-java" are easily recognized. But
> IMO, knowing whether artifacts such as "flink-ml", "flink-clients",
> "flink-table" have the version suffix or not is difficult for newcomers.
> >
> > This is why we are adding the version suffix to all Scala 2.11 artifacts
> currently. For Scala 2.10 artifacts, we aren’t adding the version suffix
> for Flink with Java users.
> >
> > I’m for adding the version suffix to Scala 2.10 artifacts also. But I’m
> not sure that removing the version suffix from Java-only artifacts would be
> good. As I said above, It seems difficult for newcomers.
> >
> > Regards,
> > Chiwan Park
> >
> > On November 2, 2015 at 8:19:15 PM, Fabian Hueske (fhueske@gmail.com)
> wrote:
> >
> > That would mean to have "flink-java_2.10" and "flink-java_2.11" artifacts
> > (and others that depend on flink-java and have no other Scala dependency)
> > in the 0.10.0 release and only "flink-java" in the next 1.0 release.
> >
> > Do we want that?
> >
> > 2015-11-02 11:37 GMT+01:00 Maximilian Michels <mxm@apache.org>:
> >
> >> I'm for leaving it as-is and renaming all artifacts which depend on
> >> Scala for the release following 0.10.
> >>
> >> On Mon, Nov 2, 2015 at 11:32 AM, Fabian Hueske <fhueske@gmail.com>
> wrote:
> >> > OK, let me try to summarize the discussion (and please correct me if I
> >> got
> >> > something wrong).
> >> >
> >> > 1) Flink deploys Scala 2.11 snapshot artifacts. Therefore, we have to
> >> > release 2.11 artifacts for the 0.10.0 release version as well.
> >> >
> >> > 2) Everybody agrees to appropriately tag all artifacts that have a
> >> > (transitive) Scala dependency. ATM, that would also include flink-java
> >> > which is a bit awkward. The Scala dependency in flink-java originates
> >> from
> >> > the Chill library which is used to obtain a Kryo serializer which is
> >> > initialized with serializers for Scala classes. We could resolve this
> >> issue
> >> > by providing Java and Scala specific implementations of the Kryo
> >> > serializers and have KryoTypeInfos for Java and Scala.
> >> >
> >> > The question to answer right now is, do we want to have "correctly"
> >> labeled
> >> > artifacts for the next 0.10.0 release or do we defer that for 1.0?
> >> > If we want to solve it for 0.10.0 we need to cancel the current RC and
> >> > provide a fix to remove the Scala dependency in flink-java, IMO.
> >> >
> >> > Opinions?
> >> >
> >> > Cheers, Fabian
> >> >
> >> > 2015-11-02 8:55 GMT+01:00 Stephan Ewen <sewen@apache.org>:
> >> >
> >> >> +1 for the approach discusses here, and for removing Scala
> dependencies
> >> >> from modules that can be Scala independent.
> >> >>
> >> >> It would be nice if pure Java users would not see any Scala
> versioning
> >> (on
> >> >> flink-core, flink-java, later also flink-sreaming-java). I guess for
> any
> >> >> runtime-related parts (including flink-client and currently all
> >> streaming
> >> >> projects), we need the Scala versions...
> >> >>
> >> >> On Sun, Nov 1, 2015 at 9:29 AM, Maximilian Michels <mxm@apache.org>
> >> wrote:
> >> >>
> >> >> > Good point. Didn't know that. We can still add them for the
> release.
> >> >> >
> >> >> > On Sat, Oct 31, 2015 at 1:51 PM, Alexander Alexandrov
> >> >> > <alexander.s.alexandrov@gmail.com> wrote:
> >> >> > > My two cents - there are already Maven artifacts deployed
for
> 2.11
> >> in
> >> >> the
> >> >> > > SNAPSHOT repository. I think it might be confusing if they
> suddenly
> >> >> > > disappear for the stable release.
> >> >> > >
> >> >> > >
> >> >> > > 2015-10-29 11:58 GMT+01:00 Maximilian Michels <mxm@apache.org>:
> >> >> > >
> >> >> > >> Seems like we agree that we need artifacts for different
> versions
> >> of
> >> >> > Scala
> >> >> > >> on Maven. There also seems to be a preference for including
the
> >> >> version
> >> >> > in
> >> >> > >> the artifact name.
> >> >> > >>
> >> >> > >> I've created an issue and marked it to be resolved for
1.0. For
> the
> >> >> 0.10
> >> >> > >> release, we will have binaries but no Maven artifacts.
The
> biggest
> >> >> > >> challenge I see is to remove Scala from as many modules
as
> >> possible.
> >> >> For
> >> >> > >> example, flink-java depends on Scala at the moment..
> >> >> > >>
> >> >> > >> https://issues.apache.org/jira/browse/FLINK-2940
> >> >> > >>
> >> >> > >> On Wed, Oct 28, 2015 at 7:31 PM, Frederick F. Kautz IV
<
> >> >> > fkautz@redhat.com>
> >> >> > >> wrote:
> >> >> > >>
> >> >> > >> > No idea if I get a vote ;) Nevertheless, +1 to have
binaries
> for
> >> >> both
> >> >> > >> > versions in Maven and explicitly "scala versioned".
> >> >> > >> >
> >> >> > >> > Some background on this for those not as familiar
with scala
> >> >> > versioning:
> >> >> > >> >
> >> >> > >> > It's considered best practice to label what version
of scala a
> >> >> library
> >> >> > >> > uses in the artifact id.
> >> >> > >> >
> >> >> > >> > The reason is compiled scala code is only compatible
with the
> >> major
> >> >> > >> > version of scala it was compiled for. For example,
a library
> >> >> > compatible
> >> >> > >> > with 2.10 is not compatible with 2.11. The same
will be true
> with
> >> >> 2.12
> >> >> > >> once
> >> >> > >> > it is released. Mixing versions will result in undefined
> behavior
> >> >> > which
> >> >> > >> > will likely manifest itself as runtime exceptions.
> >> >> > >> >
> >> >> > >> > The convention to fix this problem is for all published
> >> libraries to
> >> >> > >> > specify the version of scala they are compatible
with. Leaving
> >> out
> >> >> the
> >> >> > >> > scala version in a library is akin to saying "We
don't depend
> on
> >> >> scala
> >> >> > >> for
> >> >> > >> > this library, so feel free to use whatever you want."
Sbt
> users
> >> will
> >> >> > >> > typically specify the version of scala they use
and tooling is
> >> built
> >> >> > >> around
> >> >> > >> > ensuring consistency with the "%%" operator.
> >> >> > >> >
> >> >> > >> > E.g.
> >> >> > >> >
> >> >> > >> > scalaVersion := "2.11.4"
> >> >> > >> >
> >> >> > >> > // this resolves to to artifactID: "scalacheck_2.11"
> >> >> > >> > libraryDependencies += "org.scalacheck" %% "scalacheck"
%
> >> "1.12.0" %
> >> >> > >> "test"
> >> >> > >> >
> >> >> > >> > The most important part of this is that the scala
version is
> >> >> explicit
> >> >> > >> > which eliminates the problem for downstream users.
> >> >> > >> >
> >> >> > >> > Cheers,
> >> >> > >> > Frederick
> >> >> > >> >
> >> >> > >> >
> >> >> > >> > On 10/28/2015 10:55 AM, Fabian Hueske wrote:
> >> >> > >> >
> >> >> > >> >> +1 to have binaries for both versions in Maven
and as build
> to
> >> >> > download.
> >> >> > >> >>
> >> >> > >> >> 2015-10-26 17:11 GMT+01:00 Theodore Vasiloudis
<
> >> >> > >> >> theodoros.vasiloudis@gmail.com>:
> >> >> > >> >>
> >> >> > >> >> +1 for having binaries, I'm working on a Spark
application
> >> >> currently
> >> >> > >> with
> >> >> > >> >>> Scala 2.11 and having to rebuild everything
when deploying
> >> e.g. to
> >> >> > EC2
> >> >> > >> >>> is a
> >> >> > >> >>> pain.
> >> >> > >> >>>
> >> >> > >> >>> On Mon, Oct 26, 2015 at 4:22 PM, Ufuk Celebi
<
> uce@apache.org>
> >> >> > wrote:
> >> >> > >> >>>
> >> >> > >> >>> I agree with Till, but is this something
you want to
> address in
> >> >> this
> >> >> > >> >>>> release already?
> >> >> > >> >>>>
> >> >> > >> >>>> I would postpone it to 1.0.0.
> >> >> > >> >>>>
> >> >> > >> >>>> – Ufuk
> >> >> > >> >>>>
> >> >> > >> >>>> On 26 Oct 2015, at 16:17, Till Rohrmann
<
> trohrmann@apache.org
> >> >
> >> >> > wrote:
> >> >> > >> >>>>>
> >> >> > >> >>>>> I would be in favor of deploying
also Scala 2.11
> artifacts to
> >> >> > Maven
> >> >> > >> >>>>>
> >> >> > >> >>>> since
> >> >> > >> >>>
> >> >> > >> >>>> more and more people will try out Flink
with Scala 2.11.
> >> Having
> >> >> the
> >> >> > >> >>>>> dependencies in the Maven repository
makes it considerably
> >> >> easier
> >> >> > for
> >> >> > >> >>>>> people to get their Flink jobs running.
> >> >> > >> >>>>>
> >> >> > >> >>>>> Furthermore, I observed that people
are not aware that our
> >> >> > deployed
> >> >> > >> >>>>> artifacts, e.g. flink-runtime, are
built with Scala 2.10.
> As
> >> a
> >> >> > >> >>>>>
> >> >> > >> >>>> consequence,
> >> >> > >> >>>>
> >> >> > >> >>>>> they mix flink dependencies with
other dependencies
> pulling
> >> in
> >> >> > Scala
> >> >> > >> >>>>>
> >> >> > >> >>>> 2.11
> >> >> > >> >>>
> >> >> > >> >>>> and then they wonder that the program
crashes. It would be,
> >> imho,
> >> >> > >> >>>>>
> >> >> > >> >>>> clearer
> >> >> > >> >>>
> >> >> > >> >>>> if all our dependencies which depend
on a specific Scala
> >> version
> >> >> > would
> >> >> > >> >>>>>
> >> >> > >> >>>> have
> >> >> > >> >>>>
> >> >> > >> >>>>> the corresponding Scala suffix appended.
> >> >> > >> >>>>>
> >> >> > >> >>>>> Adding the 2.10 suffix would also
spare us the hassle of
> >> >> upgrading
> >> >> > >> to a
> >> >> > >> >>>>> newer Scala version in the future,
because then the
> artifacts
> >> >> > >> wouldn't
> >> >> > >> >>>>> share the same artifact name.
> >> >> > >> >>>>>
> >> >> > >> >>>>> Cheers,
> >> >> > >> >>>>> Till
> >> >> > >> >>>>>
> >> >> > >> >>>>> On Mon, Oct 26, 2015 at 4:04 PM,
Maximilian Michels <
> >> >> > mxm@apache.org>
> >> >> > >> >>>>>
> >> >> > >> >>>> wrote:
> >> >> > >> >>>>
> >> >> > >> >>>>> Hi Flinksters,
> >> >> > >> >>>>>>
> >> >> > >> >>>>>> We have recently committed an
easy way to change Flink's
> >> Scala
> >> >> > >> >>>>>>
> >> >> > >> >>>>> version.
> >> >> > >> >>>
> >> >> > >> >>>> The
> >> >> > >> >>>>
> >> >> > >> >>>>> question arises now whether we should
ship Scala 2.11 as
> >> >> binaries
> >> >> > and
> >> >> > >> >>>>>>
> >> >> > >> >>>>> via
> >> >> > >> >>>>
> >> >> > >> >>>>> Maven. For the rc0, I created all
binaries twice, for
> Scala
> >> 2.10
> >> >> > and
> >> >> > >> >>>>>>
> >> >> > >> >>>>> 2.11.
> >> >> > >> >>>>
> >> >> > >> >>>>> However, I didn't create Maven artifacts.
This follows our
> >> >> current
> >> >> > >> >>>>>>
> >> >> > >> >>>>> shipping
> >> >> > >> >>>>
> >> >> > >> >>>>> strategy where we only ship Hadoop1
and Hadoop 2.3.0 Maven
> >> >> > >> >>>>>>
> >> >> > >> >>>>> dependencies
> >> >> > >> >>>
> >> >> > >> >>>> but
> >> >> > >> >>>>
> >> >> > >> >>>>> additionally Hadoop 2.4, 2.6, 2.7
as binaries.
> >> >> > >> >>>>>>
> >> >> > >> >>>>>> Should we also upload Maven
dependencies for Scala 2.11?
> >> >> > >> >>>>>>
> >> >> > >> >>>>>> If so, the next question arises:
What version pattern
> >> should we
> >> >> > have
> >> >> > >> >>>>>>
> >> >> > >> >>>>> for
> >> >> > >> >>>
> >> >> > >> >>>> the Flink Scala 2.11 dependencies? For
Hadoop, we append
> >> -hadoop1
> >> >> > to
> >> >> > >> >>>>>>
> >> >> > >> >>>>> the
> >> >> > >> >>>
> >> >> > >> >>>> VERSION, e.g. artifactID=flink-core,
version=0.9.1-hadoop1.
> >> >> > >> >>>>>>
> >> >> > >> >>>>>> However, it is common practice
to append the suffix to
> the
> >> >> > >> artifactID
> >> >> > >> >>>>>>
> >> >> > >> >>>>> of
> >> >> > >> >>>
> >> >> > >> >>>> the Maven dependency, e.g. artifactID=flink-core_2.11,
> >> >> > version=0.9.1.
> >> >> > >> >>>>>>
> >> >> > >> >>>>> This
> >> >> > >> >>>>
> >> >> > >> >>>>> has mostly historic reasons but
is widely used.
> >> >> > >> >>>>>>
> >> >> > >> >>>>>> Whatever naming pattern we choose,
it should be
> consistent.
> >> I
> >> >> > would
> >> >> > >> be
> >> >> > >> >>>>>>
> >> >> > >> >>>>> in
> >> >> > >> >>>>
> >> >> > >> >>>>> favor of changing our artifact names
to contain the Hadoop
> >> and
> >> >> > Scala
> >> >> > >> >>>>>> version. This would also imply
that all Scala dependent
> >> Maven
> >> >> > >> modules
> >> >> > >> >>>>>> receive a Scala suffix (also
the default Scala 2.10
> >> modules).
> >> >> > >> >>>>>>
> >> >> > >> >>>>>> Cheers,
> >> >> > >> >>>>>> Max
> >> >> > >> >>>>>>
> >> >> > >> >>>>>>
> >> >> > >> >>>>
> >> >> > >> >
> >> >> > >>
> >> >> >
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message