flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Flink REST API async?
Date Tue, 08 Aug 2017 13:46:22 GMT
I quickly talked to Till about this. The new JobManager, once FLIP-6 is implemented, will have
a new REST endpoint that allows submitting a JobGraph directly. With this, we no longer have
to execute the user main() method in the WebRuntimeMonitor (which is a component that the
current JobManager process loads to serve the web frontend and the REST interface).

This should solve the problem, but unfortunately it doesn't solve your current problem.

Best,
Aljoscha
> On 8. Aug 2017, at 10:26, Francisco Gonzalez Barea <Francisco.Gonzalez@piksel.com>
wrote:
> 
> Aha ok… Thanks for your answer Eron.
> 
> Regards
> 
> 
>> On 7 Aug 2017, at 19:04, Eron Wright <eronwright@gmail.com <mailto:eronwright@gmail.com>>
wrote:
>> 
>> When you submit a program via the REST API, the main method executes inside the JobManager
process.    Unfortunately a static variable is used to establish the execution environment
that the program obtains from `ExecutionEnvironment.getExecutionEnvironment()`.  From the
stack trace it appears that two main methods are executing simultaneously and one is corrupting
the other.
>> 
>> On Mon, Aug 7, 2017 at 8:21 AM, Francisco Gonzalez Barea <Francisco.Gonzalez@piksel.com
<mailto:Francisco.Gonzalez@piksel.com>> wrote:
>> Hi there!
>> 
>> We are doing some POCs submitting jobs remotely to Flink. We tried with Flink CLI
and now we´re testing the Rest API. 
>> 
>> So the point is that when we try to execute a set of requests in an async way (using
CompletableFutures) only a couple of them run successfully. For the rest we get the exception
copied at the end of the email.
>> 
>> Do you know the reason for this?
>> 
>> Thanks in advance!!
>> Regards,
>> 
>> org.apache.flink.client.program.ProgramInvocationException: The main method caused
an error.
>>   at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
>>   at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
>>   at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:80)
>>  at org.apache.flink.client.program.ClusterClient.getOptimizedPlan(ClusterClient.java:318)
>>  at org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:72)
>>  at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.handleJsonRequest(JarRunHandler.java:61)
>>  at org.apache.flink.runtime.webmonitor.handlers.AbstractJsonRequestHandler.handleRequest(AbstractJsonRequestHandler.java:41)
>>  at org.apache.flink.runtime.webmonitor.RuntimeMonitorHandler.respondAsLeader(RuntimeMonitorHandler.java:109)
>>  at org.apache.flink.runtime.webmonitor.RuntimeMonitorHandlerBase.channelRead0(RuntimeMonitorHandlerBase.java:97)
>>  at org.apache.flink.runtime.webmonitor.RuntimeMonitorHandlerBase.channelRead0(RuntimeMonitorHandlerBase.java:44)
>>  at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>>  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>>  at io.netty.handler.codec.http.router.Handler.routed(Handler.java:62)
>>  at io.netty.handler.codec.http.router.DualAbstractHandler.channelRead0(DualAbstractHandler.java:57)
>>  at io.netty.handler.codec.http.router.DualAbstractHandler.channelRead0(DualAbstractHandler.java:20)
>>  at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>>  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>>  at org.apache.flink.runtime.webmonitor.HttpRequestHandler.channelRead0(HttpRequestHandler.java:159)
>>  at org.apache.flink.runtime.webmonitor.HttpRequestHandler.channelRead0(HttpRequestHandler.java:65)
>>  at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>>  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>>  at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>>  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>>  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)
>>  at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147)
>>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
>>  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
>>  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847)
>>  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
>>  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>>  at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>>  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>>  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>>  at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
>>  at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
>>  at java.lang.Thread.run(Thread.java:748)
>>  \nCaused by: java.util.ConcurrentModificationException
>>  at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
>>  at java.util.ArrayList$Itr.next(ArrayList.java:851)
>>  at org.apache.flink.api.java.operators.OperatorTranslation.translateToPlan(OperatorTranslation.java:49)
>>  at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:1065)
>>  at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:1032)
>>  at org.apache.flink.client.program.OptimizerPlanEnvironment.execute(OptimizerPlanEnvironment.java:47)
>>  at com.piksel.sequoia.media.common.launcher.Launcher.main(Launcher.java:56)
>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>  at java.lang.reflect.Method.invoke(Method.java:498)
>>  at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
>>  ... 39 more\n”}
>> 
>> 
>> This message is private and confidential. If you have received this message in error,
please notify the sender or servicedesk@piksel.com <mailto:servicedesk@piksel.com> and
remove it from your system.
>> 
>> Piksel Inc is a company registered in the United States, 2100 Powers Ferry Road SE,
Suite 400, Atlanta, GA 30339
>> 
>> 
> 


Mime
View raw message