beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From amir bahmanyari <amirto...@yahoo.com>
Subject Re: Problem with Pipeline In Flink Runner
Date Thu, 19 May 2016 23:22:16 GMT
Hi Colleagues,Pls let me know this is not related to StreamingMode so I can create a separate
thread for it.I thought it might be relevant.
I am trying to run a Storm app in a flink-1.0.0 Cluster following this link.Apache Flink
0.10.2 Documentation: Storm Compatibility

  
|  
|   
|   
|   |    |

   |

  |
|  
|   |  
Apache Flink 0.10.2 Documentation: Storm Compatibility
   |   |

  |

  |

 

I have everything i.e. fat jar cleanly built & packaged.I have not added any jars to flink's
lib folder. Just the default jars there.Following is the only dependency in pom.xml &
everything gets compiled & packaged by this dependency.<dependency> <groupId>org.apache.flink</groupId>
<artifactId>flink-storm</artifactId> <version>0.10.2</version>   
  </dependency>
When I try to run it in Flink Cluster, I get a StreamingMode related MethodNotFound exception.
I see from Maven report that flink runtime jar has been included in the packaged fat jar.
java.lang.NoSuchMethodError: org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster.<init>(Lorg/apache/flink/configuration/Configuration;ZLorg/apache/flink/runtime/StreamingMode;)V 
      at org.apache.flink.storm.api.FlinkLocalCluster.<init>(FlinkLocalCluster.java:54) 
      at com.myco.flinkstorm.SimpleTopology.main(SimpleTopology.java:102)I appreciate any
idea/suggestions pls.Thanks for your help.
Amir-      From: Aljoscha Krettek <aljoscha@apache.org>
 To: user@beam.incubator.apache.org 
 Sent: Thursday, May 19, 2016 10:31 AM
 Subject: Re: Problem with Pipeline In Flink Runner
   
Hi,as an addition, there are plans to get rid of the "streaming" flag for all runners and
determine the mode from the job.
-Aljoscha
On Thu, 19 May 2016 at 18:52 Maximilian Michels <mxm@apache.org> wrote:

Hi Ismaël,

There is a pull request which enables Windows on batch. It will be
merged soon. In the meantime, please use the streaming execution mode
of the Flink Runner.

Use setStreaming(true) to enable it on the FlinkPipelineOptions.

Cheers,
Max

On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía <iemejia@gmail.com> wrote:
> I was trying to run a local Pipeline we have (basically a batch one we
> rewrote from TPC-H)  in all the possible runners, for the moment I had
> success with the DirectPipelineRunner, InProcessPipelineRunner and
> SparkPipelineRunner. However when I executed it in the FlinkPipelineRunner I
> got this exception:
>
> java.lang.UnsupportedOperationException: The transform Window.Into() is
> currently not supported.
>
> I thought it was probably a configuration error since I remember that in the
> flink runner we had to explicitly define if it was a batch or streaming job,
> but when I checked the options:
>
> Current Settings:
>   appName: TestQ08Task
>   filesToStage: ...
>   flinkMaster: [auto]
>   parallelism: 1
>   runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
>   stableUniqueNames: WARNING
>   streaming: false
>
> I noticed I was in batch mode, is there something missing in the options
> configuration, or is it that the UnsoppertedOperation is a WIP or not
> supported at all by Flink ?
>
> And extra question, this option to define if it is in batch or stream mode
> is going to stay for long, can't be this inferred some how ?
>
> Thanks,
> -Ismaël
>



  
Mime
View raw message