flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Testing Apache Flink 0.9.0-rc1
Date Tue, 09 Jun 2015 23:32:56 GMT
While looking into FLINK-2188 (HBase input) I've discovered that Hadoop input formats implementing
Configurable (like mapreduce.TableInputFormat) don't have the Hadoop configuration set via
setConf(Configuration).

I have a small fix for this, which I have to clean up. First, I wanted to check what you think
about this issue wrt the release. Personally, I think this is a release blocker, because it
essentially means that no Hadoop input format, which relies on the Configuration instance
to be set this way will work (this is to some extent a bug of the respective input formats)
– most notably the HBase TableInputFormat.

– Ufuk

On 09 Jun 2015, at 18:07, Chiwan Park <chiwanpark@icloud.com> wrote:

> I attached jps and jstack log about hanging TaskManagerFailsWithSlotSharingITCase to
JIRA FLINK-2183.
> 
> Regards,
> Chiwan Park
> 
>> On Jun 10, 2015, at 12:28 AM, Aljoscha Krettek <aljoscha@apache.org> wrote:
>> 
>> I discovered something that might be a feature, rather than a bug. When you
>> submit an example using the web client without giving parameters the
>> program fails with this:
>> 
>> org.apache.flink.client.program.ProgramInvocationException: The main method
>> caused an error.
>> 
>> at
>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
>> 
>> at
>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
>> 
>> at org.apache.flink.client.program.Client.run(Client.java:315)
>> 
>> at
>> org.apache.flink.client.web.JobSubmissionServlet.doGet(JobSubmissionServlet.java:302)
>> 
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
>> 
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
>> 
>> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:532)
>> 
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
>> 
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227)
>> 
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:965)
>> 
>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:388)
>> 
>> at
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:187)
>> 
>> at
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:901)
>> 
>> at
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
>> 
>> at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:47)
>> 
>> at
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:113)
>> 
>> at org.eclipse.jetty.server.Server.handle(Server.java:352)
>> 
>> at
>> org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)
>> 
>> at
>> org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
>> 
>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:549)
>> 
>> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:211)
>> 
>> at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:425)
>> 
>> at
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:489)
>> 
>> at
>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
>> 
>> at java.lang.Thread.run(Thread.java:745)
>> 
>> Caused by: java.lang.NullPointerException
>> 
>> at
>> org.apache.flink.api.common.JobExecutionResult.getAccumulatorResult(JobExecutionResult.java:78)
>> 
>> at org.apache.flink.api.java.DataSet.collect(DataSet.java:409)
>> 
>> at org.apache.flink.api.java.DataSet.print(DataSet.java:1345)
>> 
>> at
>> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:80)
>> 
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> 
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> 
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> 
>> at java.lang.reflect.Method.invoke(Method.java:497)
>> 
>> at
>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
>> 
>> ... 24 more
>> 
>> 
>> This also only occurs when you uncheck the "suspend execution while showing
>> plan".
>> 
>> I think this arises because the new print() uses collect() which tries to
>> get the job execution result. I guess the result is Null since the job is
>> submitted asynchronously when the checkbox is unchecked.
>> 
>> 
>> Other than that, the new print() is pretty sweet when you run the builtin
>> examples from the CLI. You get all the state changes and also the result,
>> even when running in cluster mode on several task managers. :D
>> 
>> 
>> On Tue, Jun 9, 2015 at 3:41 PM, Aljoscha Krettek <aljoscha@apache.org>
>> wrote:
>> 
>>> I discovered another problem:
>>> https://issues.apache.org/jira/browse/FLINK-2191 The closure cleaner
>>> cannot be disabled in part of the Streaming Java API and all of the
>>> Streaming Scala API. I think this is a release blocker (in addition
>>> the the other bugs found so far.)
>>> 
>>> On Tue, Jun 9, 2015 at 2:35 PM, Aljoscha Krettek <aljoscha@apache.org>
>>> wrote:
>>>> I found the bug in the failing YARNSessionFIFOITCase: It was comparing
>>>> the hostname to a hostname in some yarn config. In one case it was
>>>> capitalised, in the other case it wasn't.
>>>> 
>>>> Pushing fix to master and release-0.9 branch.
>>>> 
>>>> On Tue, Jun 9, 2015 at 2:18 PM, Sachin Goel <sachingoel0101@gmail.com>
>>> wrote:
>>>>> A re-ran lead to reproducibility of 11 failures again.
>>>>> TaskManagerTest.testSubmitAndExecuteTask was failing with a time-out
but
>>>>> managed to succeed in a re-run. Here is the log output again:
>>>>> http://pastebin.com/raw.php?i=N4cm1J18
>>>>> 
>>>>> Setup: JDK 1.8.0_40 on windows 8.1
>>>>> System memory: 8GB, quad-core with maximum 8 threads.
>>>>> 
>>>>> Regards
>>>>> Sachin Goel
>>>>> 
>>>>> On Tue, Jun 9, 2015 at 5:34 PM, Ufuk Celebi <uce@apache.org> wrote:
>>>>> 
>>>>>> 
>>>>>> On 09 Jun 2015, at 13:58, Sachin Goel <sachingoel0101@gmail.com>
>>> wrote:
>>>>>> 
>>>>>>> On my local machine, several flink runtime tests are failing
on "mvn
>>>>>> clean
>>>>>>> verify". Here is the log output:
>>> http://pastebin.com/raw.php?i=VWbx2ppf
>>>>>> 
>>>>>> Thanks for reporting this. Have you tried it multiple times? Is it
>>> failing
>>>>>> reproducibly with the same tests? What's your setup?
>>>>>> 
>>>>>> – Ufuk
>>> 
> 
> 
> 


Mime
View raw message