flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Scala 2.10/2.11 Maven dependencies
Date Mon, 02 Nov 2015 20:01:43 GMT
+1 for that.

2015-11-02 20:52 GMT+01:00 Stephan Ewen <sewen@apache.org>:

> +1 for Max' suggestion to fix that for 1.0 (next release).
>
> Hot fixing of this thing so short before a release is a bit risky in my
> opinion. It is easy to make errors (overlooking something, error not
> visible because of cached older dependencies, ...), it happened more than
> once during version upgrades, maven project re-organizations, etc.
>
> Doing it after 0.10 and having a few week to let it sink in and errors
> surface would probably be much safer...
>
> On Mon, Nov 2, 2015 at 5:19 AM, Fabian Hueske <fhueske@gmail.com> wrote:
>
> > Ah OK. Sorry, I misunderstood your intention.
> >
> > 2015-11-02 14:07 GMT+01:00 Maximilian Michels <mxm@apache.org>:
> >
> > > > That would mean to have "flink-java_2.10" and "flink-java_2.11"
> > artifacts
> > > > (and others that depend on flink-java and have no other Scala
> > dependency)
> > > > in the 0.10.0 release and only "flink-java" in the next 1.0 release.
> > >
> > >
> > > My suggestion was to keep the Scala unsuffixed Scala 2.10 and add a
> > > suffix for Scala 2.11. That's the way we are currently doing it (also
> > > deployed on Maven like this). For the next release after 0.10, we can
> > > do it properly.
> > >
> > > On Mon, Nov 2, 2015 at 1:30 PM, Chiwan Park <chiwanpark@apache.org>
> > wrote:
> > > > If we choose selective Scala version suffix for artifacts, we have to
> > > tell which artifacts have the version suffix to newcomers. Some
> artifacts
> > > such as "flink-java”, "flink-streaming-java" are easily recognized. But
> > > IMO, knowing whether artifacts such as "flink-ml", "flink-clients",
> > > "flink-table" have the version suffix or not is difficult for
> newcomers.
> > > >
> > > > This is why we are adding the version suffix to all Scala 2.11
> > artifacts
> > > currently. For Scala 2.10 artifacts, we aren’t adding the version
> suffix
> > > for Flink with Java users.
> > > >
> > > > I’m for adding the version suffix to Scala 2.10 artifacts also. But
> I’m
> > > not sure that removing the version suffix from Java-only artifacts
> would
> > be
> > > good. As I said above, It seems difficult for newcomers.
> > > >
> > > > Regards,
> > > > Chiwan Park
> > > >
> > > > On November 2, 2015 at 8:19:15 PM, Fabian Hueske (fhueske@gmail.com)
> > > wrote:
> > > >
> > > > That would mean to have "flink-java_2.10" and "flink-java_2.11"
> > artifacts
> > > > (and others that depend on flink-java and have no other Scala
> > dependency)
> > > > in the 0.10.0 release and only "flink-java" in the next 1.0 release.
> > > >
> > > > Do we want that?
> > > >
> > > > 2015-11-02 11:37 GMT+01:00 Maximilian Michels <mxm@apache.org>:
> > > >
> > > >> I'm for leaving it as-is and renaming all artifacts which depend on
> > > >> Scala for the release following 0.10.
> > > >>
> > > >> On Mon, Nov 2, 2015 at 11:32 AM, Fabian Hueske <fhueske@gmail.com>
> > > wrote:
> > > >> > OK, let me try to summarize the discussion (and please correct
me
> > if I
> > > >> got
> > > >> > something wrong).
> > > >> >
> > > >> > 1) Flink deploys Scala 2.11 snapshot artifacts. Therefore, we
have
> > to
> > > >> > release 2.11 artifacts for the 0.10.0 release version as well.
> > > >> >
> > > >> > 2) Everybody agrees to appropriately tag all artifacts that have
a
> > > >> > (transitive) Scala dependency. ATM, that would also include
> > flink-java
> > > >> > which is a bit awkward. The Scala dependency in flink-java
> > originates
> > > >> from
> > > >> > the Chill library which is used to obtain a Kryo serializer which
> is
> > > >> > initialized with serializers for Scala classes. We could resolve
> > this
> > > >> issue
> > > >> > by providing Java and Scala specific implementations of the Kryo
> > > >> > serializers and have KryoTypeInfos for Java and Scala.
> > > >> >
> > > >> > The question to answer right now is, do we want to have
> "correctly"
> > > >> labeled
> > > >> > artifacts for the next 0.10.0 release or do we defer that for
1.0?
> > > >> > If we want to solve it for 0.10.0 we need to cancel the current
RC
> > and
> > > >> > provide a fix to remove the Scala dependency in flink-java, IMO.
> > > >> >
> > > >> > Opinions?
> > > >> >
> > > >> > Cheers, Fabian
> > > >> >
> > > >> > 2015-11-02 8:55 GMT+01:00 Stephan Ewen <sewen@apache.org>:
> > > >> >
> > > >> >> +1 for the approach discusses here, and for removing Scala
> > > dependencies
> > > >> >> from modules that can be Scala independent.
> > > >> >>
> > > >> >> It would be nice if pure Java users would not see any Scala
> > > versioning
> > > >> (on
> > > >> >> flink-core, flink-java, later also flink-sreaming-java).
I guess
> > for
> > > any
> > > >> >> runtime-related parts (including flink-client and currently
all
> > > >> streaming
> > > >> >> projects), we need the Scala versions...
> > > >> >>
> > > >> >> On Sun, Nov 1, 2015 at 9:29 AM, Maximilian Michels <
> mxm@apache.org
> > >
> > > >> wrote:
> > > >> >>
> > > >> >> > Good point. Didn't know that. We can still add them
for the
> > > release.
> > > >> >> >
> > > >> >> > On Sat, Oct 31, 2015 at 1:51 PM, Alexander Alexandrov
> > > >> >> > <alexander.s.alexandrov@gmail.com> wrote:
> > > >> >> > > My two cents - there are already Maven artifacts
deployed for
> > > 2.11
> > > >> in
> > > >> >> the
> > > >> >> > > SNAPSHOT repository. I think it might be confusing
if they
> > > suddenly
> > > >> >> > > disappear for the stable release.
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > 2015-10-29 11:58 GMT+01:00 Maximilian Michels <
> mxm@apache.org
> > >:
> > > >> >> > >
> > > >> >> > >> Seems like we agree that we need artifacts
for different
> > > versions
> > > >> of
> > > >> >> > Scala
> > > >> >> > >> on Maven. There also seems to be a preference
for including
> > the
> > > >> >> version
> > > >> >> > in
> > > >> >> > >> the artifact name.
> > > >> >> > >>
> > > >> >> > >> I've created an issue and marked it to be resolved
for 1.0.
> > For
> > > the
> > > >> >> 0.10
> > > >> >> > >> release, we will have binaries but no Maven
artifacts. The
> > > biggest
> > > >> >> > >> challenge I see is to remove Scala from as
many modules as
> > > >> possible.
> > > >> >> For
> > > >> >> > >> example, flink-java depends on Scala at the
moment..
> > > >> >> > >>
> > > >> >> > >> https://issues.apache.org/jira/browse/FLINK-2940
> > > >> >> > >>
> > > >> >> > >> On Wed, Oct 28, 2015 at 7:31 PM, Frederick
F. Kautz IV <
> > > >> >> > fkautz@redhat.com>
> > > >> >> > >> wrote:
> > > >> >> > >>
> > > >> >> > >> > No idea if I get a vote ;) Nevertheless,
+1 to have
> binaries
> > > for
> > > >> >> both
> > > >> >> > >> > versions in Maven and explicitly "scala
versioned".
> > > >> >> > >> >
> > > >> >> > >> > Some background on this for those not
as familiar with
> scala
> > > >> >> > versioning:
> > > >> >> > >> >
> > > >> >> > >> > It's considered best practice to label
what version of
> > scala a
> > > >> >> library
> > > >> >> > >> > uses in the artifact id.
> > > >> >> > >> >
> > > >> >> > >> > The reason is compiled scala code is only
compatible with
> > the
> > > >> major
> > > >> >> > >> > version of scala it was compiled for.
For example, a
> library
> > > >> >> > compatible
> > > >> >> > >> > with 2.10 is not compatible with 2.11.
The same will be
> true
> > > with
> > > >> >> 2.12
> > > >> >> > >> once
> > > >> >> > >> > it is released. Mixing versions will result
in undefined
> > > behavior
> > > >> >> > which
> > > >> >> > >> > will likely manifest itself as runtime
exceptions.
> > > >> >> > >> >
> > > >> >> > >> > The convention to fix this problem is
for all published
> > > >> libraries to
> > > >> >> > >> > specify the version of scala they are
compatible with.
> > Leaving
> > > >> out
> > > >> >> the
> > > >> >> > >> > scala version in a library is akin to
saying "We don't
> > depend
> > > on
> > > >> >> scala
> > > >> >> > >> for
> > > >> >> > >> > this library, so feel free to use whatever
you want." Sbt
> > > users
> > > >> will
> > > >> >> > >> > typically specify the version of scala
they use and
> tooling
> > is
> > > >> built
> > > >> >> > >> around
> > > >> >> > >> > ensuring consistency with the "%%" operator.
> > > >> >> > >> >
> > > >> >> > >> > E.g.
> > > >> >> > >> >
> > > >> >> > >> > scalaVersion := "2.11.4"
> > > >> >> > >> >
> > > >> >> > >> > // this resolves to to artifactID: "scalacheck_2.11"
> > > >> >> > >> > libraryDependencies += "org.scalacheck"
%% "scalacheck" %
> > > >> "1.12.0" %
> > > >> >> > >> "test"
> > > >> >> > >> >
> > > >> >> > >> > The most important part of this is that
the scala version
> is
> > > >> >> explicit
> > > >> >> > >> > which eliminates the problem for downstream
users.
> > > >> >> > >> >
> > > >> >> > >> > Cheers,
> > > >> >> > >> > Frederick
> > > >> >> > >> >
> > > >> >> > >> >
> > > >> >> > >> > On 10/28/2015 10:55 AM, Fabian Hueske
wrote:
> > > >> >> > >> >
> > > >> >> > >> >> +1 to have binaries for both versions
in Maven and as
> build
> > > to
> > > >> >> > download.
> > > >> >> > >> >>
> > > >> >> > >> >> 2015-10-26 17:11 GMT+01:00 Theodore
Vasiloudis <
> > > >> >> > >> >> theodoros.vasiloudis@gmail.com>:
> > > >> >> > >> >>
> > > >> >> > >> >> +1 for having binaries, I'm working
on a Spark
> application
> > > >> >> currently
> > > >> >> > >> with
> > > >> >> > >> >>> Scala 2.11 and having to rebuild
everything when
> deploying
> > > >> e.g. to
> > > >> >> > EC2
> > > >> >> > >> >>> is a
> > > >> >> > >> >>> pain.
> > > >> >> > >> >>>
> > > >> >> > >> >>> On Mon, Oct 26, 2015 at 4:22 PM,
Ufuk Celebi <
> > > uce@apache.org>
> > > >> >> > wrote:
> > > >> >> > >> >>>
> > > >> >> > >> >>> I agree with Till, but is this
something you want to
> > > address in
> > > >> >> this
> > > >> >> > >> >>>> release already?
> > > >> >> > >> >>>>
> > > >> >> > >> >>>> I would postpone it to 1.0.0.
> > > >> >> > >> >>>>
> > > >> >> > >> >>>> – Ufuk
> > > >> >> > >> >>>>
> > > >> >> > >> >>>> On 26 Oct 2015, at 16:17,
Till Rohrmann <
> > > trohrmann@apache.org
> > > >> >
> > > >> >> > wrote:
> > > >> >> > >> >>>>>
> > > >> >> > >> >>>>> I would be in favor of
deploying also Scala 2.11
> > > artifacts to
> > > >> >> > Maven
> > > >> >> > >> >>>>>
> > > >> >> > >> >>>> since
> > > >> >> > >> >>>
> > > >> >> > >> >>>> more and more people will
try out Flink with Scala
> 2.11.
> > > >> Having
> > > >> >> the
> > > >> >> > >> >>>>> dependencies in the Maven
repository makes it
> > considerably
> > > >> >> easier
> > > >> >> > for
> > > >> >> > >> >>>>> people to get their Flink
jobs running.
> > > >> >> > >> >>>>>
> > > >> >> > >> >>>>> Furthermore, I observed
that people are not aware that
> > our
> > > >> >> > deployed
> > > >> >> > >> >>>>> artifacts, e.g. flink-runtime,
are built with Scala
> > 2.10.
> > > As
> > > >> a
> > > >> >> > >> >>>>>
> > > >> >> > >> >>>> consequence,
> > > >> >> > >> >>>>
> > > >> >> > >> >>>>> they mix flink dependencies
with other dependencies
> > > pulling
> > > >> in
> > > >> >> > Scala
> > > >> >> > >> >>>>>
> > > >> >> > >> >>>> 2.11
> > > >> >> > >> >>>
> > > >> >> > >> >>>> and then they wonder that
the program crashes. It would
> > be,
> > > >> imho,
> > > >> >> > >> >>>>>
> > > >> >> > >> >>>> clearer
> > > >> >> > >> >>>
> > > >> >> > >> >>>> if all our dependencies which
depend on a specific
> Scala
> > > >> version
> > > >> >> > would
> > > >> >> > >> >>>>>
> > > >> >> > >> >>>> have
> > > >> >> > >> >>>>
> > > >> >> > >> >>>>> the corresponding Scala
suffix appended.
> > > >> >> > >> >>>>>
> > > >> >> > >> >>>>> Adding the 2.10 suffix
would also spare us the hassle
> of
> > > >> >> upgrading
> > > >> >> > >> to a
> > > >> >> > >> >>>>> newer Scala version in
the future, because then the
> > > artifacts
> > > >> >> > >> wouldn't
> > > >> >> > >> >>>>> share the same artifact
name.
> > > >> >> > >> >>>>>
> > > >> >> > >> >>>>> Cheers,
> > > >> >> > >> >>>>> Till
> > > >> >> > >> >>>>>
> > > >> >> > >> >>>>> On Mon, Oct 26, 2015 at
4:04 PM, Maximilian Michels <
> > > >> >> > mxm@apache.org>
> > > >> >> > >> >>>>>
> > > >> >> > >> >>>> wrote:
> > > >> >> > >> >>>>
> > > >> >> > >> >>>>> Hi Flinksters,
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>>> We have recently committed
an easy way to change
> > Flink's
> > > >> Scala
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>> version.
> > > >> >> > >> >>>
> > > >> >> > >> >>>> The
> > > >> >> > >> >>>>
> > > >> >> > >> >>>>> question arises now whether
we should ship Scala 2.11
> as
> > > >> >> binaries
> > > >> >> > and
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>> via
> > > >> >> > >> >>>>
> > > >> >> > >> >>>>> Maven. For the rc0, I
created all binaries twice, for
> > > Scala
> > > >> 2.10
> > > >> >> > and
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>> 2.11.
> > > >> >> > >> >>>>
> > > >> >> > >> >>>>> However, I didn't create
Maven artifacts. This follows
> > our
> > > >> >> current
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>> shipping
> > > >> >> > >> >>>>
> > > >> >> > >> >>>>> strategy where we only
ship Hadoop1 and Hadoop 2.3.0
> > Maven
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>> dependencies
> > > >> >> > >> >>>
> > > >> >> > >> >>>> but
> > > >> >> > >> >>>>
> > > >> >> > >> >>>>> additionally Hadoop 2.4,
2.6, 2.7 as binaries.
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>>> Should we also upload
Maven dependencies for Scala
> > 2.11?
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>>> If so, the next question
arises: What version pattern
> > > >> should we
> > > >> >> > have
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>> for
> > > >> >> > >> >>>
> > > >> >> > >> >>>> the Flink Scala 2.11 dependencies?
For Hadoop, we
> append
> > > >> -hadoop1
> > > >> >> > to
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>> the
> > > >> >> > >> >>>
> > > >> >> > >> >>>> VERSION, e.g. artifactID=flink-core,
> > version=0.9.1-hadoop1.
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>>> However, it is common
practice to append the suffix
> to
> > > the
> > > >> >> > >> artifactID
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>> of
> > > >> >> > >> >>>
> > > >> >> > >> >>>> the Maven dependency, e.g.
artifactID=flink-core_2.11,
> > > >> >> > version=0.9.1.
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>> This
> > > >> >> > >> >>>>
> > > >> >> > >> >>>>> has mostly historic reasons
but is widely used.
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>>> Whatever naming pattern
we choose, it should be
> > > consistent.
> > > >> I
> > > >> >> > would
> > > >> >> > >> be
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>> in
> > > >> >> > >> >>>>
> > > >> >> > >> >>>>> favor of changing our
artifact names to contain the
> > Hadoop
> > > >> and
> > > >> >> > Scala
> > > >> >> > >> >>>>>> version. This would
also imply that all Scala
> dependent
> > > >> Maven
> > > >> >> > >> modules
> > > >> >> > >> >>>>>> receive a Scala suffix
(also the default Scala 2.10
> > > >> modules).
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>>> Cheers,
> > > >> >> > >> >>>>>> Max
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>>>
> > > >> >> > >> >>>>
> > > >> >> > >> >
> > > >> >> > >>
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message