flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: [DISCUSS] Release 1.3.0 RC0 (Non voting, testing release candidate)
Date Mon, 15 May 2017 14:06:51 GMT
I'll now kick off the RC creation process.

On Sun, May 14, 2017 at 7:14 PM, Stephan Ewen <sewen@apache.org> wrote:

> +1 for another RC once the shading fix is merged.
>
> Even if RC1 would have pending issues, users that want to help testing need
> a new RC, because (as Robert said), RC0's shading issues make it virtually
> unusable...
>
> On Fri, May 12, 2017 at 3:45 PM, Robert Metzger <rmetzger@apache.org>
> wrote:
>
> > I'm considering to do another non voting RC1 on Monday with the
> incremental
> > checkpointing and maven fixes in (and of course as many other things as
> > possible). Once the other critical fixes are in (ideally Tuesday or
> > Wednesday), I'll create the first voting one.
> >
> > The current RC0 is almost unusable because of the incorrect shading.
> >
> > On Fri, May 12, 2017 at 2:09 PM, Greg Hogan <code@greghogan.com> wrote:
> >
> > > +1 for sticking to the code freeze deadline and building a new release
> > > candidate but since the release is still two weeks off (5/26) I think
> it
> > > better to delay voting to give time for additional bug fixes.
> > >
> > >
> > > > On May 11, 2017, at 10:19 AM, Robert Metzger <rmetzger@apache.org>
> > > wrote:
> > > >
> > > > It seems that we found quite a large number of critical issues in the
> > > first
> > > > RC.
> > > >
> > > > - FLINK-6537 Umbrella issue for fixes to incremental snapshots
> (Stefan
> > > has
> > > > a PR open to fix the critical ones)
> > > > - FLINK-6531 Deserialize checkpoint hooks with user classloader (has
> a
> > > > pending PR)
> > > > - FLINK-6515 KafkaConsumer checkpointing fails because of ClassLoader
> > > > issues (status unknown)
> > > > - FLINK-6514 Cannot start Flink Cluster in standalone mode (Stephan
> is
> > on
> > > > it)
> > > > - FLINK-6508 Include license files of packaged dependencies (Stephan
> is
> > > on
> > > > it) + FLINK-6501 Make sure NOTICE files are bundled into shaded JAR
> > files
> > > > - FLINK-6284 Incorrect sorting of completed checkpoints in
> > > > ZooKeeperCompletedCheckpointStore (unknown)
> > > >
> > > > I would like to get these issues fixed by end of this week (Sunday),
> so
> > > > that I can create the first voting RC on Monday morning.
> > > > Please reject if you think we will not manage to get the stuff fixed
> > > until
> > > > then.
> > > >
> > > >
> > > >
> > > > On Thu, May 11, 2017 at 10:54 AM, Till Rohrmann <
> trohrmann@apache.org>
> > > > wrote:
> > > >
> > > >> Unfortunately, it won't be fully functional in 1.3.
> > > >>
> > > >> Cheers,
> > > >> Till
> > > >>
> > > >> On Thu, May 11, 2017 at 10:45 AM, Renjie Liu <
> liurenjie2008@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >>> @Rohrmann Will FLIP 6 be fully functional in 1.3 release?
> > > >>>
> > > >>> On Thu, May 11, 2017 at 4:12 PM Gyula Fóra <gyula.fora@gmail.com>
> > > wrote:
> > > >>>
> > > >>>> Thanks Stefan!
> > > >>>> Gyula
> > > >>>>
> > > >>>> Stefan Richter <s.richter@data-artisans.com> ezt írta
(időpont:
> > 2017.
> > > >>> máj.
> > > >>>> 11., Cs, 10:04):
> > > >>>>
> > > >>>>>
> > > >>>>> Hi,
> > > >>>>>
> > > >>>>> Thanks for reporting this. I found a couple of issues
yesterday
> > and I
> > > >>> am
> > > >>>>> currently working on a bundle of fixes. I will take a
look at
> this
> > > >>>> problem,
> > > >>>>> and if it is already covered.
> > > >>>>>
> > > >>>>> Best,
> > > >>>>> Stefan
> > > >>>>>
> > > >>>>>> Am 11.05.2017 um 09:47 schrieb Gyula Fóra <gyula.fora@gmail.com
> >:
> > > >>>>>>
> > > >>>>>> Hi,
> > > >>>>>> I am not sure if this belong to this thread, but while
trying to
> > > >> run
> > > >>> a
> > > >>>>> job
> > > >>>>>> with rocks incremental backend I ran into 2 issues:
> > > >>>>>>
> > > >>>>>> One with savepoints, I can't figure out because I
can't make
> sense
> > > >> of
> > > >>>> the
> > > >>>>>> error or how it happenned:
> > > >>>>>> The error stack trace is here:
> > > >>>>>> https://gist.github.com/gyfora/2f7bb387bbd9f455f9702908cde0b239
> > > >>>>>> This happens on every savepoint attempt and seems
to be related
> to
> > > >>> the
> > > >>>>>> kafka source. Interestingly other tasks succeed in
writing data
> to
> > > >>>> hdfs.
> > > >>>>>>
> > > >>>>>> The other one is covered by
> > > >>>>> https://issues.apache.org/jira/browse/FLINK-6531 I
> > > >>>>>> guess. I am not sure if the first one is related though.
> > > >>>>>>
> > > >>>>>> Thank you!
> > > >>>>>> Gyula
> > > >>>>>>
> > > >>>>>> Till Rohrmann <trohrmann@apache.org> ezt írta
(időpont: 2017.
> > máj.
> > > >>>> 11.,
> > > >>>>> Cs,
> > > >>>>>> 9:14):
> > > >>>>>>
> > > >>>>>>> Hi Renjie,
> > > >>>>>>>
> > > >>>>>>> 1.3 already contains some Flip-6 code. However,
it is not yet
> > > >> fully
> > > >>>>>>> functional. What you already can do is to run
local jobs on the
> > > >>> Flip-6
> > > >>>>> code
> > > >>>>>>> base by instantiating a MiniCluster and then using
the
> > > >>>>>>> Flip6LocalStreamEnvironment.
> > > >>>>>>>
> > > >>>>>>> Cheers,
> > > >>>>>>> Till
> > > >>>>>>>
> > > >>>>>>> On Thu, May 11, 2017 at 6:00 AM, Renjie Liu <
> > > >>> liurenjie2008@gmail.com>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Hi, all:
> > > >>>>>>>> Will the FLIP 6 be included in release 1.3?
> > > >>>>>>>>
> > > >>>>>>>> On Wed, May 10, 2017 at 9:48 PM Gyula Fóra
<
> > gyula.fora@gmail.com
> > > >>>
> > > >>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Thanks you! :)
> > > >>>>>>>>>
> > > >>>>>>>>> Chesnay Schepler <chesnay@apache.org>
ezt írta (időpont:
> 2017.
> > > >>> máj.
> > > >>>>>>> 10.,
> > > >>>>>>>>> Sze, 15:44):
> > > >>>>>>>>>
> > > >>>>>>>>>> I guess it is related to this one
> > > >>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-6514
?
> > > >>>>>>>>>>
> > > >>>>>>>>>> On 10.05.2017 15:34, Gyula Fóra wrote:
> > > >>>>>>>>>>> Hi,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I tried to run an application
on 1.3 but I keep getting the
> > > >>>>>>> following
> > > >>>>>>>>>> error:
> > > >>>>>>>>>>> java.lang.NoClassDefFoundError:
Could not initialize class
> > > >>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation
> > > >>>>>>>>>>> at
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>> org.apache.flink.runtime.security.modules.HadoopModule.
> > > >>>>>>>> install(HadoopModule.java:45)
> > > >>>>>>>>>>> at
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>> org.apache.flink.runtime.security.SecurityUtils.
> > > >>>>>>>> install(SecurityUtils.java:78)
> > > >>>>>>>>>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.
> > > >>> java:1128)
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Even after adding hadoop-common
to the lib manually (which
> I
> > > >>> guess
> > > >>>>>>>>> should
> > > >>>>>>>>>>> not be necessary).
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Any idea what might cause this?
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Thanks,
> > > >>>>>>>>>>> Gyula
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Chesnay Schepler <chesnay@apache.org>
ezt írta (időpont:
> > > >> 2017.
> > > >>>>>>> máj.
> > > >>>>>>>>> 9.,
> > > >>>>>>>>>> K,
> > > >>>>>>>>>>> 13:27):
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> I looked into AppVeyor and
managed to create a working
> setup
> > > >>> that
> > > >>>>>>>>>>>> replicates the problems i
encounter locally.
> > > >>>>>>>>>>>> https://ci.appveyor.com/project/zentol/flink/build/1.0.12
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> The integration should be
as simple as placing an
> > > >> appveyor.yml
> > > >>>>>>>>> (similar
> > > >>>>>>>>>>>> to the travis.yml) into the
repository and asking INFRA to
> > > >>>> enabled
> > > >>>>>>>> it
> > > >>>>>>>>>>>> for Flink.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> I've opened a JIRA to track
this initiative:
> > > >>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-6510
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On 09.05.2017 10:17, Robert
Metzger wrote:
> > > >>>>>>>>>>>>> Chesnay, I would propose
to look into a CI solution for
> > > >>> Windows.
> > > >>>>>>>> I'll
> > > >>>>>>>>>> try
> > > >>>>>>>>>>>>> out appveyor.com for my
Flink fork and report back how
> > well
> > > >>> it
> > > >>>>>>>> works
> > > >>>>>>>>>>>> (let
> > > >>>>>>>>>>>>> me know if you want to
look into it).
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Tue, May 9, 2017 at
10:05 AM, Stephan Ewen <
> > > >>> sewen@apache.org
> > > >>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> @Renjie What functionality
are you looking for? Most
> stuff
> > > >>>>>>> should
> > > >>>>>>>>> work
> > > >>>>>>>>>>>> in
> > > >>>>>>>>>>>>>> the normal master
branch.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Tue, May 9, 2017
at 10:02 AM, Renjie Liu <
> > > >>>>>>>>> liurenjie2008@gmail.com>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Stephan:
> > > >>>>>>>>>>>>>>> This means that
 we can use FLIP 6 for basic
> > functionality
> > > >>>> now?
> > > >>>>>>>> Any
> > > >>>>>>>>>>>>>>> document for setting
up that? Will FLIP 6 be finished
> in
> > > >>>>>>> release
> > > >>>>>>>>> 1.3?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On Tue, May 9,
2017 at 3:43 PM Stephan Ewen <
> > > >>> sewen@apache.org
> > > >>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> @Renjie Liu:
A good part of the FLIP-6 code is in
> there,
> > > >>> but
> > > >>>>>>> not
> > > >>>>>>>>>> all.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> It does run
well on Yarn, Mesos, Docker, etc.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> We need to
finish the FLIP-6 work mainly for full
> > > >>> elasticity.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On Tue, May
9, 2017 at 5:24 AM, Renjie Liu <
> > > >>>>>>>>> liurenjie2008@gmail.com
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> Hi, does
this include the FLIP6?
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Tue,
May 9, 2017 at 2:29 AM Stephan Ewen <
> > > >>>>>>> sewen@apache.org>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>> Did
a quick test: Simply adding the
> > > >>>>>>> "org.apache.maven.plugins.
> > > >>>>>>>>>>>>>>>>>> shade.resource.ApacheNoticeResourceTransformer"
> helps
> > > >>> with
> > > >>>>>>>>> NOTICE
> > > >>>>>>>>>>>>>>>>>> files,
> > > >>>>>>>>>>>>>>>>>> but
does not add the required BSD licence copies.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> On
Mon, May 8, 2017 at 8:25 PM, Stephan Ewen <
> > > >>>>>>>> sewen@apache.org>
> > > >>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>>
I did the first pass for the legal check.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
  - Source LICENSE and NOTICE are okay
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
  - In the shaded JAR files, we are not bundling
> the
> > > >>>>>>>> license
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>>
notice files of the dependencies we include in the
> > > >>> shaded
> > > >>>>>>>> jars.
> > > >>>>>>>>>>>>>>>>>>>
     => Not a problem for Guava (Apache Licensed)
> > > >>>>>>>>>>>>>>>>>>>
     => I think is a problem for ASM
> (redistribution
> > > >> in
> > > >>>>>>>>> binary
> > > >>>>>>>>>>>> form,
> > > >>>>>>>>>>>>>>>>>>>
hence needs a notice of the copy)
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
  - The Table API / SQL module needs more entries
> for
> > > >>>>>>>> Janino
> > > >>>>>>>>> /
> > > >>>>>>>>>>>>>>>>>>>
Reflections (both BSD licensed)
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
So that is definitely a blocker.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
On Mon, May 8, 2017 at 12:14 PM, Robert Metzger <
> > > >>>>>>>>>>>> rmetzger@apache.org
> > > >>>>>>>>>>>>>>>>>>>
wrote:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
Hi Devs,
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
I've created a first non-voting release candidate
> > for
> > > >>>>>>> Flink
> > > >>>>>>>>>> 1.3.0.
> > > >>>>>>>>>>>>>>>>>>>>
Please use this RC to test as much as you can and
> > > >>> provide
> > > >>>>>>>>>> feedback
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>>>>>>>>
the Flink community. The more we find and fix now,
> > > >> the
> > > >>>>>>>> better
> > > >>>>>>>>>>>> Flink
> > > >>>>>>>>>>>>>> 1.3.0
> > > >>>>>>>>>>>>>>>>>>>>
wil be :)
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
I've CC'ed the user@ mailing list to get more
> > people
> > > >>> to
> > > >>>>>>>> test
> > > >>>>>>>>>> it.
> > > >>>>>>>>>>>> DO
> > > >>>>>>>>>>>>>>>>>>>>
NOT USE THIS RELEASE CANDIDATE IN PRODUCTION.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
I will prepare a google document to synchronize
> the
> > > >>>>>>> testing
> > > >>>>>>>>>>>> effort a
> > > >>>>>>>>>>>>>>>>>>>>
bit more.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
Depending on the number of issues we identify, I
> > hope
> > > >>>> that
> > > >>>>>>>> we
> > > >>>>>>>>>> can
> > > >>>>>>>>>>>> do
> > > >>>>>>>>>>>>>>>>>>>>
the first VOTEing RC early next week.
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
------------------------------
> > > >>>>>>>> ------------------------------
> > > >>>>>>>>>>>>>> ---------
> > > >>>>>>>>>>>>>>>>>>>>
The release commit is
> > > >>>>>>>>> f94c002991dcce9f1104f8e61b43efb2f8247cb4,
> > > >>>>>>>>>>>>>>>>>>>>
located here:
> > > >>>>>>> http://git-wip-us.apache.org/repos/asf/flink/
> > > >>>>>>>>>>>>>>>>>>>>
commit/f94c0029
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
The artifacts are located here:
> > http://people.apache
> > > >> .
> > > >>>>>>>>>>>>>>>>>>>>
org/~rmetzger/flink-1.3.0-rc0/
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
The maven staging repository is here:
> > > >>> https://repository
> > > >>>> .
> > > >>>>>>>>>>>>>>>>>>>>
apache.org/content/repositorie
> s/orgapacheflink-1118
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
------------------------------
> > > >>>>>>>> ------------------------------
> > > >>>>>>>>>>>>>> ---------
> > > >>>>>>>>>>>>>>>>>>>>
Happy testing!
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
Regards,
> > > >>>>>>>>>>>>>>>>>>>>
Robert
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>>>>> Liu, Renjie
> > > >>>>>>>>>>>>>>>>> Software
Engineer, MVAD
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>>> Liu, Renjie
> > > >>>>>>>>>>>>>>> Software Engineer,
MVAD
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>> --
> > > >>>>>>>> Liu, Renjie
> > > >>>>>>>> Software Engineer, MVAD
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>> --
> > > >>> Liu, Renjie
> > > >>> Software Engineer, MVAD
> > > >>>
> > > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message