flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@informatik.hu-berlin.de>
Subject Re: question on flink-storm-examples
Date Tue, 01 Sep 2015 20:33:40 GMT
Yes. That is what I expected.

JobManager cannot start the job, due to less task slots. It logs the
exception NoResourceAvailableException (it is not shown in stdout; see
"log" folder). There is no feedback to Flink CLI that the job could not
be started.

Furthermore, WordCount-StormTopology sleeps for 5 seconds and tries to
"kill" the job. However, because the job was never started, there is a
NotAliveException which in print to stdout.

-Matthias



On 09/01/2015 10:26 PM, Jerry Peng wrote:
> When I run WordCount-StormTopology I get the following exception:
> 
> ~/flink/bin/flink run WordCount-StormTopology.jar
> hdfs:///home/jerrypeng/hadoop/hadoop_dir/data/data.txt
> hdfs:///home/jerrypeng/hadoop/hadoop_dir/data/results.txt
> 
> org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error.
> 
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:452)
> 
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353)
> 
> at org.apache.flink.client.program.Client.run(Client.java:278)
> 
> at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:631)
> 
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:319)
> 
> at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:954)
> 
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1004)
> 
> Caused by: NotAliveException(msg:null)
> 
> at
> org.apache.flink.stormcompatibility.api.FlinkClient.killTopologyWithOpts(FlinkClient.java:209)
> 
> at
> org.apache.flink.stormcompatibility.api.FlinkClient.killTopology(FlinkClient.java:203)
> 
> at
> org.apache.flink.stormcompatibility.wordcount.StormWordCountRemoteBySubmitter.main(StormWordCountRemoteBySubmitter.java:80)
> 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 
> at java.lang.reflect.Method.invoke(Method.java:483)
> 
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437)
> 
> ... 6 more
> 
> 
> The exception above occurred while trying to run your command.
> 
> 
> Any idea how to fix this?
> 
> On Tue, Sep 1, 2015 at 3:10 PM, Matthias J. Sax
> <mjsax@informatik.hu-berlin.de <mailto:mjsax@informatik.hu-berlin.de>>
> wrote:
> 
>     Hi Jerry,
> 
>     WordCount-StormTopology uses a hard coded dop of 4. If you start up
>     Flink in local mode (bin/start-local-streaming.sh), you need to increase
>     the number of task slots to at least 4 in conf/flink-conf.yaml before
>     starting Flink -> taskmanager.numberOfTaskSlots
> 
>     You should actually see the following exception in
>     log/flink-...-jobmanager-...log
> 
>     > NoResourceAvailableException: Not enough free slots available to
>     run the job. You can decrease the operator parallelism or increase
>     the number of slots per TaskManager in the configuration.
> 
>     WordCount-StormTopology does use StormWordCountRemoteBySubmitter
>     internally. So, you do use it already ;)
> 
>     I am not sure what you mean by "get rid of KafkaSource"? It is still in
>     the code base. Which version to you use? In flink-0.10-SNAPSHOT it is
>     located in submodule "flink-connector-kafka" (which is submodule of
>     "flink-streaming-connector-parent" -- which is submodule of
>     "flink-streamping-parent").
> 
> 
>     -Matthias
> 
> 
>     On 09/01/2015 09:40 PM, Jerry Peng wrote:
>     > Hello,
>     >
>     > I have some questions regarding how to run one of the
>     > flink-storm-examples, the WordCountTopology.  How should I run the
>     job?
>     > On github its says I should just execute
>     > bin/flink run example.jar but when I execute:
>     >
>     > bin/flink run WordCount-StormTopology.jar
>     >
>     > nothing happens.  What am I doing wrong? and How can I run the
>     > WordCounttopology via StormWordCountRemoteBySubmitter?
>     >
>     > Also why did you guys get rid of the KafkaSource class?  What is
>     the API
>     > now for subscribing to a kafka source?
>     >
>     > Best,
>     >
>     > Jerry
> 
> 


Mime
View raw message