flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: [DISCUSS] Release 1.3.0 RC0 (Non voting, testing release candidate)
Date Sun, 14 May 2017 17:14:59 GMT
+1 for another RC once the shading fix is merged.

Even if RC1 would have pending issues, users that want to help testing need
a new RC, because (as Robert said), RC0's shading issues make it virtually
unusable...

On Fri, May 12, 2017 at 3:45 PM, Robert Metzger <rmetzger@apache.org> wrote:

> I'm considering to do another non voting RC1 on Monday with the incremental
> checkpointing and maven fixes in (and of course as many other things as
> possible). Once the other critical fixes are in (ideally Tuesday or
> Wednesday), I'll create the first voting one.
>
> The current RC0 is almost unusable because of the incorrect shading.
>
> On Fri, May 12, 2017 at 2:09 PM, Greg Hogan <code@greghogan.com> wrote:
>
> > +1 for sticking to the code freeze deadline and building a new release
> > candidate but since the release is still two weeks off (5/26) I think it
> > better to delay voting to give time for additional bug fixes.
> >
> >
> > > On May 11, 2017, at 10:19 AM, Robert Metzger <rmetzger@apache.org>
> > wrote:
> > >
> > > It seems that we found quite a large number of critical issues in the
> > first
> > > RC.
> > >
> > > - FLINK-6537 Umbrella issue for fixes to incremental snapshots (Stefan
> > has
> > > a PR open to fix the critical ones)
> > > - FLINK-6531 Deserialize checkpoint hooks with user classloader (has a
> > > pending PR)
> > > - FLINK-6515 KafkaConsumer checkpointing fails because of ClassLoader
> > > issues (status unknown)
> > > - FLINK-6514 Cannot start Flink Cluster in standalone mode (Stephan is
> on
> > > it)
> > > - FLINK-6508 Include license files of packaged dependencies (Stephan is
> > on
> > > it) + FLINK-6501 Make sure NOTICE files are bundled into shaded JAR
> files
> > > - FLINK-6284 Incorrect sorting of completed checkpoints in
> > > ZooKeeperCompletedCheckpointStore (unknown)
> > >
> > > I would like to get these issues fixed by end of this week (Sunday), so
> > > that I can create the first voting RC on Monday morning.
> > > Please reject if you think we will not manage to get the stuff fixed
> > until
> > > then.
> > >
> > >
> > >
> > > On Thu, May 11, 2017 at 10:54 AM, Till Rohrmann <trohrmann@apache.org>
> > > wrote:
> > >
> > >> Unfortunately, it won't be fully functional in 1.3.
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Thu, May 11, 2017 at 10:45 AM, Renjie Liu <liurenjie2008@gmail.com
> >
> > >> wrote:
> > >>
> > >>> @Rohrmann Will FLIP 6 be fully functional in 1.3 release?
> > >>>
> > >>> On Thu, May 11, 2017 at 4:12 PM Gyula Fóra <gyula.fora@gmail.com>
> > wrote:
> > >>>
> > >>>> Thanks Stefan!
> > >>>> Gyula
> > >>>>
> > >>>> Stefan Richter <s.richter@data-artisans.com> ezt írta (időpont:
> 2017.
> > >>> máj.
> > >>>> 11., Cs, 10:04):
> > >>>>
> > >>>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> Thanks for reporting this. I found a couple of issues yesterday
> and I
> > >>> am
> > >>>>> currently working on a bundle of fixes. I will take a look
at this
> > >>>> problem,
> > >>>>> and if it is already covered.
> > >>>>>
> > >>>>> Best,
> > >>>>> Stefan
> > >>>>>
> > >>>>>> Am 11.05.2017 um 09:47 schrieb Gyula Fóra <gyula.fora@gmail.com>:
> > >>>>>>
> > >>>>>> Hi,
> > >>>>>> I am not sure if this belong to this thread, but while
trying to
> > >> run
> > >>> a
> > >>>>> job
> > >>>>>> with rocks incremental backend I ran into 2 issues:
> > >>>>>>
> > >>>>>> One with savepoints, I can't figure out because I can't
make sense
> > >> of
> > >>>> the
> > >>>>>> error or how it happenned:
> > >>>>>> The error stack trace is here:
> > >>>>>> https://gist.github.com/gyfora/2f7bb387bbd9f455f9702908cde0b239
> > >>>>>> This happens on every savepoint attempt and seems to be
related to
> > >>> the
> > >>>>>> kafka source. Interestingly other tasks succeed in writing
data to
> > >>>> hdfs.
> > >>>>>>
> > >>>>>> The other one is covered by
> > >>>>> https://issues.apache.org/jira/browse/FLINK-6531 I
> > >>>>>> guess. I am not sure if the first one is related though.
> > >>>>>>
> > >>>>>> Thank you!
> > >>>>>> Gyula
> > >>>>>>
> > >>>>>> Till Rohrmann <trohrmann@apache.org> ezt írta (időpont:
2017.
> máj.
> > >>>> 11.,
> > >>>>> Cs,
> > >>>>>> 9:14):
> > >>>>>>
> > >>>>>>> Hi Renjie,
> > >>>>>>>
> > >>>>>>> 1.3 already contains some Flip-6 code. However, it
is not yet
> > >> fully
> > >>>>>>> functional. What you already can do is to run local
jobs on the
> > >>> Flip-6
> > >>>>> code
> > >>>>>>> base by instantiating a MiniCluster and then using
the
> > >>>>>>> Flip6LocalStreamEnvironment.
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Till
> > >>>>>>>
> > >>>>>>> On Thu, May 11, 2017 at 6:00 AM, Renjie Liu <
> > >>> liurenjie2008@gmail.com>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hi, all:
> > >>>>>>>> Will the FLIP 6 be included in release 1.3?
> > >>>>>>>>
> > >>>>>>>> On Wed, May 10, 2017 at 9:48 PM Gyula Fóra <
> gyula.fora@gmail.com
> > >>>
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Thanks you! :)
> > >>>>>>>>>
> > >>>>>>>>> Chesnay Schepler <chesnay@apache.org>
ezt írta (időpont: 2017.
> > >>> máj.
> > >>>>>>> 10.,
> > >>>>>>>>> Sze, 15:44):
> > >>>>>>>>>
> > >>>>>>>>>> I guess it is related to this one
> > >>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-6514
?
> > >>>>>>>>>>
> > >>>>>>>>>> On 10.05.2017 15:34, Gyula Fóra wrote:
> > >>>>>>>>>>> Hi,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I tried to run an application on 1.3
but I keep getting the
> > >>>>>>> following
> > >>>>>>>>>> error:
> > >>>>>>>>>>> java.lang.NoClassDefFoundError: Could
not initialize class
> > >>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>> org.apache.flink.runtime.security.modules.HadoopModule.
> > >>>>>>>> install(HadoopModule.java:45)
> > >>>>>>>>>>> at
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>> org.apache.flink.runtime.security.SecurityUtils.
> > >>>>>>>> install(SecurityUtils.java:78)
> > >>>>>>>>>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.
> > >>> java:1128)
> > >>>>>>>>>>>
> > >>>>>>>>>>> Even after adding hadoop-common to
the lib manually (which I
> > >>> guess
> > >>>>>>>>> should
> > >>>>>>>>>>> not be necessary).
> > >>>>>>>>>>>
> > >>>>>>>>>>> Any idea what might cause this?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>> Gyula
> > >>>>>>>>>>>
> > >>>>>>>>>>> Chesnay Schepler <chesnay@apache.org>
ezt írta (időpont:
> > >> 2017.
> > >>>>>>> máj.
> > >>>>>>>>> 9.,
> > >>>>>>>>>> K,
> > >>>>>>>>>>> 13:27):
> > >>>>>>>>>>>
> > >>>>>>>>>>>> I looked into AppVeyor and managed
to create a working setup
> > >>> that
> > >>>>>>>>>>>> replicates the problems i encounter
locally.
> > >>>>>>>>>>>> https://ci.appveyor.com/project/zentol/flink/build/1.0.12
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> The integration should be as simple
as placing an
> > >> appveyor.yml
> > >>>>>>>>> (similar
> > >>>>>>>>>>>> to the travis.yml) into the repository
and asking INFRA to
> > >>>> enabled
> > >>>>>>>> it
> > >>>>>>>>>>>> for Flink.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> I've opened a JIRA to track this
initiative:
> > >>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-6510
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On 09.05.2017 10:17, Robert Metzger
wrote:
> > >>>>>>>>>>>>> Chesnay, I would propose to
look into a CI solution for
> > >>> Windows.
> > >>>>>>>> I'll
> > >>>>>>>>>> try
> > >>>>>>>>>>>>> out appveyor.com for my Flink
fork and report back how
> well
> > >>> it
> > >>>>>>>> works
> > >>>>>>>>>>>> (let
> > >>>>>>>>>>>>> me know if you want to look
into it).
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Tue, May 9, 2017 at 10:05
AM, Stephan Ewen <
> > >>> sewen@apache.org
> > >>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> @Renjie What functionality
are you looking for? Most stuff
> > >>>>>>> should
> > >>>>>>>>> work
> > >>>>>>>>>>>> in
> > >>>>>>>>>>>>>> the normal master branch.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Tue, May 9, 2017 at
10:02 AM, Renjie Liu <
> > >>>>>>>>> liurenjie2008@gmail.com>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Stephan:
> > >>>>>>>>>>>>>>> This means that  we
can use FLIP 6 for basic
> functionality
> > >>>> now?
> > >>>>>>>> Any
> > >>>>>>>>>>>>>>> document for setting
up that? Will FLIP 6 be finished in
> > >>>>>>> release
> > >>>>>>>>> 1.3?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Tue, May 9, 2017
at 3:43 PM Stephan Ewen <
> > >>> sewen@apache.org
> > >>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> @Renjie Liu: A
good part of the FLIP-6 code is in there,
> > >>> but
> > >>>>>>> not
> > >>>>>>>>>> all.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> It does run well
on Yarn, Mesos, Docker, etc.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> We need to finish
the FLIP-6 work mainly for full
> > >>> elasticity.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Tue, May 9,
2017 at 5:24 AM, Renjie Liu <
> > >>>>>>>>> liurenjie2008@gmail.com
> > >>>>>>>>>>>
> > >>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Hi, does this
include the FLIP6?
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Tue, May
9, 2017 at 2:29 AM Stephan Ewen <
> > >>>>>>> sewen@apache.org>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>> Did a quick
test: Simply adding the
> > >>>>>>> "org.apache.maven.plugins.
> > >>>>>>>>>>>>>>>>>> shade.resource.ApacheNoticeResourceTransformer"
helps
> > >>> with
> > >>>>>>>>> NOTICE
> > >>>>>>>>>>>>>>>>>> files,
> > >>>>>>>>>>>>>>>>>> but does
not add the required BSD licence copies.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> On Mon,
May 8, 2017 at 8:25 PM, Stephan Ewen <
> > >>>>>>>> sewen@apache.org>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>> I did
the first pass for the legal check.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>   -
Source LICENSE and NOTICE are okay
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>   -
In the shaded JAR files, we are not bundling the
> > >>>>>>>> license
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>> notice
files of the dependencies we include in the
> > >>> shaded
> > >>>>>>>> jars.
> > >>>>>>>>>>>>>>>>>>>   
  => Not a problem for Guava (Apache Licensed)
> > >>>>>>>>>>>>>>>>>>>   
  => I think is a problem for ASM (redistribution
> > >> in
> > >>>>>>>>> binary
> > >>>>>>>>>>>> form,
> > >>>>>>>>>>>>>>>>>>> hence
needs a notice of the copy)
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>   -
The Table API / SQL module needs more entries for
> > >>>>>>>> Janino
> > >>>>>>>>> /
> > >>>>>>>>>>>>>>>>>>> Reflections
(both BSD licensed)
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> So
that is definitely a blocker.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On
Mon, May 8, 2017 at 12:14 PM, Robert Metzger <
> > >>>>>>>>>>>> rmetzger@apache.org
> > >>>>>>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
Hi Devs,
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
I've created a first non-voting release candidate
> for
> > >>>>>>> Flink
> > >>>>>>>>>> 1.3.0.
> > >>>>>>>>>>>>>>>>>>>>
Please use this RC to test as much as you can and
> > >>> provide
> > >>>>>>>>>> feedback
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>>>>>>>>
the Flink community. The more we find and fix now,
> > >> the
> > >>>>>>>> better
> > >>>>>>>>>>>> Flink
> > >>>>>>>>>>>>>> 1.3.0
> > >>>>>>>>>>>>>>>>>>>>
wil be :)
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
I've CC'ed the user@ mailing list to get more
> people
> > >>> to
> > >>>>>>>> test
> > >>>>>>>>>> it.
> > >>>>>>>>>>>> DO
> > >>>>>>>>>>>>>>>>>>>>
NOT USE THIS RELEASE CANDIDATE IN PRODUCTION.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
I will prepare a google document to synchronize the
> > >>>>>>> testing
> > >>>>>>>>>>>> effort a
> > >>>>>>>>>>>>>>>>>>>>
bit more.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
Depending on the number of issues we identify, I
> hope
> > >>>> that
> > >>>>>>>> we
> > >>>>>>>>>> can
> > >>>>>>>>>>>> do
> > >>>>>>>>>>>>>>>>>>>>
the first VOTEing RC early next week.
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
------------------------------
> > >>>>>>>> ------------------------------
> > >>>>>>>>>>>>>> ---------
> > >>>>>>>>>>>>>>>>>>>>
The release commit is
> > >>>>>>>>> f94c002991dcce9f1104f8e61b43efb2f8247cb4,
> > >>>>>>>>>>>>>>>>>>>>
located here:
> > >>>>>>> http://git-wip-us.apache.org/repos/asf/flink/
> > >>>>>>>>>>>>>>>>>>>>
commit/f94c0029
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
The artifacts are located here:
> http://people.apache
> > >> .
> > >>>>>>>>>>>>>>>>>>>>
org/~rmetzger/flink-1.3.0-rc0/
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
The maven staging repository is here:
> > >>> https://repository
> > >>>> .
> > >>>>>>>>>>>>>>>>>>>>
apache.org/content/repositories/orgapacheflink-1118
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
------------------------------
> > >>>>>>>> ------------------------------
> > >>>>>>>>>>>>>> ---------
> > >>>>>>>>>>>>>>>>>>>>
Happy testing!
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
Regards,
> > >>>>>>>>>>>>>>>>>>>>
Robert
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>> Liu, Renjie
> > >>>>>>>>>>>>>>>>> Software Engineer,
MVAD
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>> Liu, Renjie
> > >>>>>>>>>>>>>>> Software Engineer,
MVAD
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> Liu, Renjie
> > >>>>>>>> Software Engineer, MVAD
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>> --
> > >>> Liu, Renjie
> > >>> Software Engineer, MVAD
> > >>>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message