Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 03C42200C74 for ; Sun, 14 May 2017 19:15:18 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 023DB160BB7; Sun, 14 May 2017 17:15:18 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id EE486160BA3 for ; Sun, 14 May 2017 19:15:16 +0200 (CEST) Received: (qmail 51054 invoked by uid 500); 14 May 2017 17:15:16 -0000 Mailing-List: contact dev-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list dev@flink.apache.org Received: (qmail 51043 invoked by uid 99); 14 May 2017 17:15:15 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 14 May 2017 17:15:15 +0000 Received: from mail-pf0-f176.google.com (mail-pf0-f176.google.com [209.85.192.176]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id B88211A003E for ; Sun, 14 May 2017 17:15:15 +0000 (UTC) Received: by mail-pf0-f176.google.com with SMTP id e193so51625077pfh.0 for ; Sun, 14 May 2017 10:15:15 -0700 (PDT) X-Gm-Message-State: AODbwcA84/Gp0ulL7oPethCF2bOA5ATEGJyHYU1WX1LJyGu+In6RkPKy 3n4fjGDPQgMQ1m0+dVWsjF4qvDa18g== X-Received: by 10.98.223.195 with SMTP id d64mr2192416pfl.226.1494782115175; Sun, 14 May 2017 10:15:15 -0700 (PDT) MIME-Version: 1.0 Received: by 10.100.169.10 with HTTP; Sun, 14 May 2017 10:14:59 -0700 (PDT) In-Reply-To: References: <565c0c92-468f-4d95-fe6a-cefe9722d18c@apache.org> <27a7c055-0756-cb4d-ea6f-0303cbe63135@apache.org> <22DF1AEB-D43B-42D0-BEF9-83216028BEB9@greghogan.com> From: Stephan Ewen Date: Sun, 14 May 2017 19:14:59 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [DISCUSS] Release 1.3.0 RC0 (Non voting, testing release candidate) To: "dev@flink.apache.org" Content-Type: multipart/alternative; boundary="94eb2c144690fde8bc054f7f14dd" archived-at: Sun, 14 May 2017 17:15:18 -0000 --94eb2c144690fde8bc054f7f14dd Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable +1 for another RC once the shading fix is merged. Even if RC1 would have pending issues, users that want to help testing need a new RC, because (as Robert said), RC0's shading issues make it virtually unusable... On Fri, May 12, 2017 at 3:45 PM, Robert Metzger wrote= : > I'm considering to do another non voting RC1 on Monday with the increment= al > checkpointing and maven fixes in (and of course as many other things as > possible). Once the other critical fixes are in (ideally Tuesday or > Wednesday), I'll create the first voting one. > > The current RC0 is almost unusable because of the incorrect shading. > > On Fri, May 12, 2017 at 2:09 PM, Greg Hogan wrote: > > > +1 for sticking to the code freeze deadline and building a new release > > candidate but since the release is still two weeks off (5/26) I think i= t > > better to delay voting to give time for additional bug fixes. > > > > > > > On May 11, 2017, at 10:19 AM, Robert Metzger > > wrote: > > > > > > It seems that we found quite a large number of critical issues in the > > first > > > RC. > > > > > > - FLINK-6537 Umbrella issue for fixes to incremental snapshots (Stefa= n > > has > > > a PR open to fix the critical ones) > > > - FLINK-6531 Deserialize checkpoint hooks with user classloader (has = a > > > pending PR) > > > - FLINK-6515 KafkaConsumer checkpointing fails because of ClassLoader > > > issues (status unknown) > > > - FLINK-6514 Cannot start Flink Cluster in standalone mode (Stephan i= s > on > > > it) > > > - FLINK-6508 Include license files of packaged dependencies (Stephan = is > > on > > > it) + FLINK-6501 Make sure NOTICE files are bundled into shaded JAR > files > > > - FLINK-6284 Incorrect sorting of completed checkpoints in > > > ZooKeeperCompletedCheckpointStore (unknown) > > > > > > I would like to get these issues fixed by end of this week (Sunday), = so > > > that I can create the first voting RC on Monday morning. > > > Please reject if you think we will not manage to get the stuff fixed > > until > > > then. > > > > > > > > > > > > On Thu, May 11, 2017 at 10:54 AM, Till Rohrmann > > > wrote: > > > > > >> Unfortunately, it won't be fully functional in 1.3. > > >> > > >> Cheers, > > >> Till > > >> > > >> On Thu, May 11, 2017 at 10:45 AM, Renjie Liu > > > >> wrote: > > >> > > >>> @Rohrmann Will FLIP 6 be fully functional in 1.3 release? > > >>> > > >>> On Thu, May 11, 2017 at 4:12 PM Gyula F=C3=B3ra > > wrote: > > >>> > > >>>> Thanks Stefan! > > >>>> Gyula > > >>>> > > >>>> Stefan Richter ezt =C3=ADrta (id=C5= =91pont: > 2017. > > >>> m=C3=A1j. > > >>>> 11., Cs, 10:04): > > >>>> > > >>>>> > > >>>>> Hi, > > >>>>> > > >>>>> Thanks for reporting this. I found a couple of issues yesterday > and I > > >>> am > > >>>>> currently working on a bundle of fixes. I will take a look at thi= s > > >>>> problem, > > >>>>> and if it is already covered. > > >>>>> > > >>>>> Best, > > >>>>> Stefan > > >>>>> > > >>>>>> Am 11.05.2017 um 09:47 schrieb Gyula F=C3=B3ra : > > >>>>>> > > >>>>>> Hi, > > >>>>>> I am not sure if this belong to this thread, but while trying to > > >> run > > >>> a > > >>>>> job > > >>>>>> with rocks incremental backend I ran into 2 issues: > > >>>>>> > > >>>>>> One with savepoints, I can't figure out because I can't make sen= se > > >> of > > >>>> the > > >>>>>> error or how it happenned: > > >>>>>> The error stack trace is here: > > >>>>>> https://gist.github.com/gyfora/2f7bb387bbd9f455f9702908cde0b239 > > >>>>>> This happens on every savepoint attempt and seems to be related = to > > >>> the > > >>>>>> kafka source. Interestingly other tasks succeed in writing data = to > > >>>> hdfs. > > >>>>>> > > >>>>>> The other one is covered by > > >>>>> https://issues.apache.org/jira/browse/FLINK-6531 I > > >>>>>> guess. I am not sure if the first one is related though. > > >>>>>> > > >>>>>> Thank you! > > >>>>>> Gyula > > >>>>>> > > >>>>>> Till Rohrmann ezt =C3=ADrta (id=C5=91pont= : 2017. > m=C3=A1j. > > >>>> 11., > > >>>>> Cs, > > >>>>>> 9:14): > > >>>>>> > > >>>>>>> Hi Renjie, > > >>>>>>> > > >>>>>>> 1.3 already contains some Flip-6 code. However, it is not yet > > >> fully > > >>>>>>> functional. What you already can do is to run local jobs on the > > >>> Flip-6 > > >>>>> code > > >>>>>>> base by instantiating a MiniCluster and then using the > > >>>>>>> Flip6LocalStreamEnvironment. > > >>>>>>> > > >>>>>>> Cheers, > > >>>>>>> Till > > >>>>>>> > > >>>>>>> On Thu, May 11, 2017 at 6:00 AM, Renjie Liu < > > >>> liurenjie2008@gmail.com> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>>> Hi, all: > > >>>>>>>> Will the FLIP 6 be included in release 1.3? > > >>>>>>>> > > >>>>>>>> On Wed, May 10, 2017 at 9:48 PM Gyula F=C3=B3ra < > gyula.fora@gmail.com > > >>> > > >>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Thanks you! :) > > >>>>>>>>> > > >>>>>>>>> Chesnay Schepler ezt =C3=ADrta (id=C5=91= pont: 2017. > > >>> m=C3=A1j. > > >>>>>>> 10., > > >>>>>>>>> Sze, 15:44): > > >>>>>>>>> > > >>>>>>>>>> I guess it is related to this one > > >>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-6514 ? > > >>>>>>>>>> > > >>>>>>>>>> On 10.05.2017 15:34, Gyula F=C3=B3ra wrote: > > >>>>>>>>>>> Hi, > > >>>>>>>>>>> > > >>>>>>>>>>> I tried to run an application on 1.3 but I keep getting the > > >>>>>>> following > > >>>>>>>>>> error: > > >>>>>>>>>>> java.lang.NoClassDefFoundError: Could not initialize class > > >>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation > > >>>>>>>>>>> at > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> org.apache.flink.runtime.security.modules.HadoopModule. > > >>>>>>>> install(HadoopModule.java:45) > > >>>>>>>>>>> at > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> org.apache.flink.runtime.security.SecurityUtils. > > >>>>>>>> install(SecurityUtils.java:78) > > >>>>>>>>>>> at org.apache.flink.client.CliFrontend.main(CliFrontend. > > >>> java:1128) > > >>>>>>>>>>> > > >>>>>>>>>>> Even after adding hadoop-common to the lib manually (which = I > > >>> guess > > >>>>>>>>> should > > >>>>>>>>>>> not be necessary). > > >>>>>>>>>>> > > >>>>>>>>>>> Any idea what might cause this? > > >>>>>>>>>>> > > >>>>>>>>>>> Thanks, > > >>>>>>>>>>> Gyula > > >>>>>>>>>>> > > >>>>>>>>>>> Chesnay Schepler ezt =C3=ADrta (id=C5= =91pont: > > >> 2017. > > >>>>>>> m=C3=A1j. > > >>>>>>>>> 9., > > >>>>>>>>>> K, > > >>>>>>>>>>> 13:27): > > >>>>>>>>>>> > > >>>>>>>>>>>> I looked into AppVeyor and managed to create a working set= up > > >>> that > > >>>>>>>>>>>> replicates the problems i encounter locally. > > >>>>>>>>>>>> https://ci.appveyor.com/project/zentol/flink/build/1.0.12 > > >>>>>>>>>>>> > > >>>>>>>>>>>> The integration should be as simple as placing an > > >> appveyor.yml > > >>>>>>>>> (similar > > >>>>>>>>>>>> to the travis.yml) into the repository and asking INFRA to > > >>>> enabled > > >>>>>>>> it > > >>>>>>>>>>>> for Flink. > > >>>>>>>>>>>> > > >>>>>>>>>>>> I've opened a JIRA to track this initiative: > > >>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-6510 > > >>>>>>>>>>>> > > >>>>>>>>>>>> On 09.05.2017 10:17, Robert Metzger wrote: > > >>>>>>>>>>>>> Chesnay, I would propose to look into a CI solution for > > >>> Windows. > > >>>>>>>> I'll > > >>>>>>>>>> try > > >>>>>>>>>>>>> out appveyor.com for my Flink fork and report back how > well > > >>> it > > >>>>>>>> works > > >>>>>>>>>>>> (let > > >>>>>>>>>>>>> me know if you want to look into it). > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On Tue, May 9, 2017 at 10:05 AM, Stephan Ewen < > > >>> sewen@apache.org > > >>>>> > > >>>>>>>>>> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> @Renjie What functionality are you looking for? Most stu= ff > > >>>>>>> should > > >>>>>>>>> work > > >>>>>>>>>>>> in > > >>>>>>>>>>>>>> the normal master branch. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> On Tue, May 9, 2017 at 10:02 AM, Renjie Liu < > > >>>>>>>>> liurenjie2008@gmail.com> > > >>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Stephan: > > >>>>>>>>>>>>>>> This means that we can use FLIP 6 for basic > functionality > > >>>> now? > > >>>>>>>> Any > > >>>>>>>>>>>>>>> document for setting up that? Will FLIP 6 be finished i= n > > >>>>>>> release > > >>>>>>>>> 1.3? > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> On Tue, May 9, 2017 at 3:43 PM Stephan Ewen < > > >>> sewen@apache.org > > >>>>> > > >>>>>>>>>> wrote: > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> @Renjie Liu: A good part of the FLIP-6 code is in ther= e, > > >>> but > > >>>>>>> not > > >>>>>>>>>> all. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> It does run well on Yarn, Mesos, Docker, etc. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> We need to finish the FLIP-6 work mainly for full > > >>> elasticity. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> On Tue, May 9, 2017 at 5:24 AM, Renjie Liu < > > >>>>>>>>> liurenjie2008@gmail.com > > >>>>>>>>>>> > > >>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Hi, does this include the FLIP6? > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> On Tue, May 9, 2017 at 2:29 AM Stephan Ewen < > > >>>>>>> sewen@apache.org> > > >>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>> Did a quick test: Simply adding the > > >>>>>>> "org.apache.maven.plugins. > > >>>>>>>>>>>>>>>>>> shade.resource.ApacheNoticeResourceTransformer" help= s > > >>> with > > >>>>>>>>> NOTICE > > >>>>>>>>>>>>>>>>>> files, > > >>>>>>>>>>>>>>>>>> but does not add the required BSD licence copies. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> On Mon, May 8, 2017 at 8:25 PM, Stephan Ewen < > > >>>>>>>> sewen@apache.org> > > >>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>> I did the first pass for the legal check. > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> - Source LICENSE and NOTICE are okay > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> - In the shaded JAR files, we are not bundling th= e > > >>>>>>>> license > > >>>>>>>>>> and > > >>>>>>>>>>>>>>>>>>> notice files of the dependencies we include in the > > >>> shaded > > >>>>>>>> jars. > > >>>>>>>>>>>>>>>>>>> =3D> Not a problem for Guava (Apache Licensed) > > >>>>>>>>>>>>>>>>>>> =3D> I think is a problem for ASM (redistribut= ion > > >> in > > >>>>>>>>> binary > > >>>>>>>>>>>> form, > > >>>>>>>>>>>>>>>>>>> hence needs a notice of the copy) > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> - The Table API / SQL module needs more entries f= or > > >>>>>>>> Janino > > >>>>>>>>> / > > >>>>>>>>>>>>>>>>>>> Reflections (both BSD licensed) > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> So that is definitely a blocker. > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> On Mon, May 8, 2017 at 12:14 PM, Robert Metzger < > > >>>>>>>>>>>> rmetzger@apache.org > > >>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Hi Devs, > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> I've created a first non-voting release candidate > for > > >>>>>>> Flink > > >>>>>>>>>> 1.3.0. > > >>>>>>>>>>>>>>>>>>>> Please use this RC to test as much as you can and > > >>> provide > > >>>>>>>>>> feedback > > >>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>>>> the Flink community. The more we find and fix now, > > >> the > > >>>>>>>> better > > >>>>>>>>>>>> Flink > > >>>>>>>>>>>>>> 1.3.0 > > >>>>>>>>>>>>>>>>>>>> wil be :) > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> I've CC'ed the user@ mailing list to get more > people > > >>> to > > >>>>>>>> test > > >>>>>>>>>> it. > > >>>>>>>>>>>> DO > > >>>>>>>>>>>>>>>>>>>> NOT USE THIS RELEASE CANDIDATE IN PRODUCTION. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> I will prepare a google document to synchronize th= e > > >>>>>>> testing > > >>>>>>>>>>>> effort a > > >>>>>>>>>>>>>>>>>>>> bit more. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Depending on the number of issues we identify, I > hope > > >>>> that > > >>>>>>>> we > > >>>>>>>>>> can > > >>>>>>>>>>>> do > > >>>>>>>>>>>>>>>>>>>> the first VOTEing RC early next week. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> ------------------------------ > > >>>>>>>> ------------------------------ > > >>>>>>>>>>>>>> --------- > > >>>>>>>>>>>>>>>>>>>> The release commit is > > >>>>>>>>> f94c002991dcce9f1104f8e61b43efb2f8247cb4, > > >>>>>>>>>>>>>>>>>>>> located here: > > >>>>>>> http://git-wip-us.apache.org/repos/asf/flink/ > > >>>>>>>>>>>>>>>>>>>> commit/f94c0029 > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> The artifacts are located here: > http://people.apache > > >> . > > >>>>>>>>>>>>>>>>>>>> org/~rmetzger/flink-1.3.0-rc0/ > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> The maven staging repository is here: > > >>> https://repository > > >>>> . > > >>>>>>>>>>>>>>>>>>>> apache.org/content/repositories/orgapacheflink-111= 8 > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> ------------------------------ > > >>>>>>>> ------------------------------ > > >>>>>>>>>>>>>> --------- > > >>>>>>>>>>>>>>>>>>>> Happy testing! > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Regards, > > >>>>>>>>>>>>>>>>>>>> Robert > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>>> Liu, Renjie > > >>>>>>>>>>>>>>>>> Software Engineer, MVAD > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>> Liu, Renjie > > >>>>>>>>>>>>>>> Software Engineer, MVAD > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> -- > > >>>>>>>> Liu, Renjie > > >>>>>>>> Software Engineer, MVAD > > >>>>>>>> > > >>>>>>> > > >>>>> > > >>>>> > > >>>> > > >>> -- > > >>> Liu, Renjie > > >>> Software Engineer, MVAD > > >>> > > >> > > > > > --94eb2c144690fde8bc054f7f14dd--