Return-Path: X-Original-To: apmail-flink-dev-archive@www.apache.org Delivered-To: apmail-flink-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 53CC918A98 for ; Wed, 10 Jun 2015 09:02:44 +0000 (UTC) Received: (qmail 39723 invoked by uid 500); 10 Jun 2015 09:02:44 -0000 Delivered-To: apmail-flink-dev-archive@flink.apache.org Received: (qmail 39659 invoked by uid 500); 10 Jun 2015 09:02:44 -0000 Mailing-List: contact dev-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list dev@flink.apache.org Received: (qmail 39648 invoked by uid 99); 10 Jun 2015 09:02:44 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Jun 2015 09:02:44 +0000 Received: from mail-lb0-f181.google.com (mail-lb0-f181.google.com [209.85.217.181]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 4BFD61A010F for ; Wed, 10 Jun 2015 09:02:43 +0000 (UTC) Received: by lbcue7 with SMTP id ue7so24710085lbc.0 for ; Wed, 10 Jun 2015 02:02:41 -0700 (PDT) X-Gm-Message-State: ALoCoQlN59cpJsZv1u9jUwG3oijJmmSaKFgDd6Fy++9UNRDG+8YC2FgUtypeZfTa2JB1c0UBgI/P X-Received: by 10.152.87.33 with SMTP id u1mr2325187laz.35.1433926961833; Wed, 10 Jun 2015 02:02:41 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.31.15 with HTTP; Wed, 10 Jun 2015 02:02:21 -0700 (PDT) In-Reply-To: References: <5B425CB2-544B-481C-8437-8BABB03F4A8B@icloud.com> <863324CF-051C-4DB3-BDF9-4BC4EFCEB759@apache.org> <86B96B16-4CBD-4F5F-A346-B74D5557DA46@icloud.com> From: Maximilian Michels Date: Wed, 10 Jun 2015 11:02:21 +0200 Message-ID: Subject: Re: Testing Apache Flink 0.9.0-rc1 To: dev@flink.apache.org Content-Type: multipart/alternative; boundary=001a11c242ea31c926051826239c --001a11c242ea31c926051826239c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I'm not against including the feature but I'd like to discuss it first. I believe that only very carefully selected commits should be added to release-0.9. If that feature happens to be tested extensively and is very important for user satisfactory then we might include it. On Wed, Jun 10, 2015 at 10:59 AM, F. Beligianni wrote: > I agree with Gyula regarding the iteration partitioning. > I have also been using this feature for developing machine learning > algorithms. And I think SAMOA also needs this feature. > > Faye > > 2015-06-10 10:54 GMT+02:00 Gyula F=C3=B3ra : > > > This feature needs to be included in the release, it has been tested an= d > > used extensively. And many applciations depend on it. > > > > Maximilian Michels ezt =C3=ADrta (id=C5=91pont: 2015. = j=C3=BAn. 10., > > Sze, > > 10:47): > > > > > With all the issues discovered, it looks like we'll have another > release > > > candidate. Right now, we have discovered the following problems: > > > > > > 1 YARN ITCase fails [fixed via 2eb5cfe] > > > 2 No Jar for SessionWindowing example [fixed in #809] > > > 3 Wrong description of the input format for the graph examples (eg. > > > ConnectedComponents) [fixed in #809] > > > 4 TaskManagerFailsWithSlotSharingITCase fails > > > 5 ComplexIntegrationTest.complexIntegrationTest1() (FLINK-2192) fails > > > 6 Submitting KMeans example to Web Submission Client does not work on > > > Firefox. > > > 7 Zooming is buggy in Web Submission Client (Firefox) > > > > > > Do we have someone familiar with the web interface who could take a > look > > at > > > the Firefox issues? > > > > > > One more important thing: The release-0.9 branch should only be used > for > > > bug fixes or prior discussed feature changes. Adding new features > defies > > > the purpose of carefully testing in advance and can have unforeseeabl= e > > > consequences. In particular, I'm referring to #810 pull request: > > > https://github.com/apache/flink/pull/810 > > > > > > IMHO, this one shouldn't have been cherry-picked onto the release-0.9 > > > branch. I would like to remove it from there if no objections are > raised. > > > > > > > > > > > > https://github.com/apache/flink/commit/e0e6f59f309170e5217bdfbf5d30db87c9= 47f8ce > > > > > > On Wed, Jun 10, 2015 at 8:52 AM, Aljoscha Krettek > > > > wrote: > > > > > > > This doesn't look good, yes. > > > > > > > > On Wed, Jun 10, 2015 at 1:32 AM, Ufuk Celebi wrote= : > > > > > > > > > While looking into FLINK-2188 (HBase input) I've discovered that > > Hadoop > > > > > input formats implementing Configurable (like > > > mapreduce.TableInputFormat) > > > > > don't have the Hadoop configuration set via setConf(Configuration= ). > > > > > > > > > > I have a small fix for this, which I have to clean up. First, I > > wanted > > > to > > > > > check what you think about this issue wrt the release. Personally= , > I > > > > think > > > > > this is a release blocker, because it essentially means that no > > Hadoop > > > > > input format, which relies on the Configuration instance to be se= t > > this > > > > way > > > > > will work (this is to some extent a bug of the respective input > > > formats) > > > > =E2=80=93 > > > > > most notably the HBase TableInputFormat. > > > > > > > > > > =E2=80=93 Ufuk > > > > > > > > > > On 09 Jun 2015, at 18:07, Chiwan Park > wrote: > > > > > > > > > > > I attached jps and jstack log about hanging > > > > > TaskManagerFailsWithSlotSharingITCase to JIRA FLINK-2183. > > > > > > > > > > > > Regards, > > > > > > Chiwan Park > > > > > > > > > > > >> On Jun 10, 2015, at 12:28 AM, Aljoscha Krettek < > > aljoscha@apache.org > > > > > > > > > wrote: > > > > > >> > > > > > >> I discovered something that might be a feature, rather than a > bug. > > > > When > > > > > you > > > > > >> submit an example using the web client without giving paramete= rs > > the > > > > > >> program fails with this: > > > > > >> > > > > > >> org.apache.flink.client.program.ProgramInvocationException: Th= e > > main > > > > > method > > > > > >> caused an error. > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedPr= ogram.java:452) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForE= xecution(PackagedProgram.java:353) > > > > > >> > > > > > >> at org.apache.flink.client.program.Client.run(Client.java:315) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServl= et.java:302) > > > > > >> > > > > > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:668= ) > > > > > >> > > > > > >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770= ) > > > > > >> > > > > > >> at > > > > > > > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453= ) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.j= ava:227) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.j= ava:965) > > > > > >> > > > > > >> at > > > > > > > > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.ja= va:187) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.ja= va:901) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:= 117) > > > > > >> > > > > > >> at > > > > > > > > > org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.jav= a:113) > > > > > >> > > > > > >> at org.eclipse.jetty.server.Server.handle(Server.java:352) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java= :596) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(Htt= pConnection.java:1048) > > > > > >> > > > > > >> at > > org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549) > > > > > >> > > > > > >> at > > > > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:21= 1) > > > > > >> > > > > > >> at > > > > > > > org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.= java:489) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.jav= a:436) > > > > > >> > > > > > >> at java.lang.Thread.run(Thread.java:745) > > > > > >> > > > > > >> Caused by: java.lang.NullPointerException > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.api.common.JobExecutionResult.getAccumulatorResult(JobEx= ecutionResult.java:78) > > > > > >> > > > > > >> at org.apache.flink.api.java.DataSet.collect(DataSet.java:409) > > > > > >> > > > > > >> at org.apache.flink.api.java.DataSet.print(DataSet.java:1345) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:80= ) > > > > > >> > > > > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java= :62) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:43) > > > > > >> > > > > > >> at java.lang.reflect.Method.invoke(Method.java:497) > > > > > >> > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedPr= ogram.java:437) > > > > > >> > > > > > >> ... 24 more > > > > > >> > > > > > >> > > > > > >> This also only occurs when you uncheck the "suspend execution > > while > > > > > showing > > > > > >> plan". > > > > > >> > > > > > >> I think this arises because the new print() uses collect() whi= ch > > > tries > > > > > to > > > > > >> get the job execution result. I guess the result is Null since > the > > > job > > > > > is > > > > > >> submitted asynchronously when the checkbox is unchecked. > > > > > >> > > > > > >> > > > > > >> Other than that, the new print() is pretty sweet when you run > the > > > > > builtin > > > > > >> examples from the CLI. You get all the state changes and also > the > > > > > result, > > > > > >> even when running in cluster mode on several task managers. :D > > > > > >> > > > > > >> > > > > > >> On Tue, Jun 9, 2015 at 3:41 PM, Aljoscha Krettek < > > > aljoscha@apache.org > > > > > > > > > > >> wrote: > > > > > >> > > > > > >>> I discovered another problem: > > > > > >>> https://issues.apache.org/jira/browse/FLINK-2191 The closure > > > cleaner > > > > > >>> cannot be disabled in part of the Streaming Java API and all = of > > the > > > > > >>> Streaming Scala API. I think this is a release blocker (in > > addition > > > > > >>> the the other bugs found so far.) > > > > > >>> > > > > > >>> On Tue, Jun 9, 2015 at 2:35 PM, Aljoscha Krettek < > > > > aljoscha@apache.org> > > > > > >>> wrote: > > > > > >>>> I found the bug in the failing YARNSessionFIFOITCase: It was > > > > comparing > > > > > >>>> the hostname to a hostname in some yarn config. In one case = it > > was > > > > > >>>> capitalised, in the other case it wasn't. > > > > > >>>> > > > > > >>>> Pushing fix to master and release-0.9 branch. > > > > > >>>> > > > > > >>>> On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel < > > > > sachingoel0101@gmail.com > > > > > > > > > > > >>> wrote: > > > > > >>>>> A re-ran lead to reproducibility of 11 failures again. > > > > > >>>>> TaskManagerTest.testSubmitAndExecuteTask was failing with a > > > > time-out > > > > > but > > > > > >>>>> managed to succeed in a re-run. Here is the log output agai= n: > > > > > >>>>> http://pastebin.com/raw.php?i=3DN4cm1J18 > > > > > >>>>> > > > > > >>>>> Setup: JDK 1.8.0_40 on windows 8.1 > > > > > >>>>> System memory: 8GB, quad-core with maximum 8 threads. > > > > > >>>>> > > > > > >>>>> Regards > > > > > >>>>> Sachin Goel > > > > > >>>>> > > > > > >>>>> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi > > > > wrote: > > > > > >>>>> > > > > > >>>>>> > > > > > >>>>>> On 09 Jun 2015, at 13:58, Sachin Goel < > > sachingoel0101@gmail.com > > > > > > > > > >>> wrote: > > > > > >>>>>> > > > > > >>>>>>> On my local machine, several flink runtime tests are > failing > > on > > > > > "mvn > > > > > >>>>>> clean > > > > > >>>>>>> verify". Here is the log output: > > > > > >>> http://pastebin.com/raw.php?i=3DVWbx2ppf > > > > > >>>>>> > > > > > >>>>>> Thanks for reporting this. Have you tried it multiple time= s? > > Is > > > it > > > > > >>> failing > > > > > >>>>>> reproducibly with the same tests? What's your setup? > > > > > >>>>>> > > > > > >>>>>> =E2=80=93 Ufuk > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --001a11c242ea31c926051826239c--