flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Looking for instructions & source for flink-java-examples-0.6-incubating-WebLogAnalysis.jar
Date Wed, 24 Sep 2014 16:43:11 GMT
I get a similar error when I use a negative value for the default
parallelism in the flink-config.yaml

I'll push a patch that gives better error messages and sanity checks the
config value, prints a warning, and falls back to DOP 1 if the config is
invalid and no manual value is provided.

Stephan


On Wed, Sep 24, 2014 at 5:48 PM, rmetzger0 <rmetzger@apache.org> wrote:

> Hi,
>
> this looks like a minor issue in your flink configuration.
> Can you have a look into your config (in the conf/flink-conf.yaml file)
> and tell me the value of the "parallelization.degree.default" property?
> If it is set to 0 or a negative value, thats the source of the issue. Set
> it to something higher (I recommend numOfNodes*numOfCoresPerNode).
>
> But I have to admit that our error reporting here is not very helpful,
> we'll improve that...
>
>
> Cheers,
> Robert
>
>
> On Wed, Sep 24, 2014 at 4:23 PM, nirvanesque [via Apache Flink (Incubator)
> User Mailing List archive.] <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=103&i=0>> wrote:
>
>> Hello robert,
>>
>> Apologies for getting back to you on this one :-(
>> Es tut mir wirklich Leid!
>>
>> I've been trying today several times in vain this programme, and
>> following is the error output :
>>
>> [QUOTE]
>> abasu@edel-4:~$ ./flink/bin/flink run -v
>> flink/examples/flink-java-examples-0.6-incubating-WebLogAnalysis.jar
>> file:///home/abasu/examples/Weblogs/documents
>> file:///home/abasu/examples/Weblogs/ranks
>> file:///home/abasu/examples/Weblogs/visits
>> file:///home/abasu/examples/Weblogs/results
>> Error: The main method caused an error.
>> org.apache.flink.client.program.ProgramInvocationException: The main
>> method caused an error.
>>         at
>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:404)
>>         at
>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:307)
>>         at org.apache.flink.client.program.Client.run(Client.java:244)
>>         at
>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:332)
>>         at org.apache.flink.client.CliFrontend.run(CliFrontend.java:319)
>>         at
>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:930)
>>         at org.apache.flink.client.CliFrontend.main(CliFrontend.java:954)
>> Caused by: java.lang.IllegalArgumentException
>>         at
>> org.apache.flink.compiler.dag.OptimizerNode.setDegreeOfParallelism(OptimizerNode.java:414)
>>         at
>> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.preVisit(PactCompiler.java:772)
>>         at
>> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.preVisit(PactCompiler.java:616)
>>         at
>> org.apache.flink.api.common.operators.base.GenericDataSinkBase.accept(GenericDataSinkBase.java:287)
>>         at org.apache.flink.api.common.Plan.accept(Plan.java:281)
>>         at
>> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:511)
>>         at
>> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:460)
>>         at
>> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:196)
>>         at
>> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:209)
>>         at org.apache.flink.client.program.Client.run(Client.java:285)
>>         at
>> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:54)
>>         at
>> org.apache.flink.example.java.relational.WebLogAnalysis.main(WebLogAnalysis.java:148)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>         at
>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:389)
>>         ... 6 more
>> abasu@edel-4:~$
>>
>> [/QUOTE]
>>
>>
>> I'm using Flink version 0.6 on a cluster of 10 nodes.
>> Job launched from Job manager node.
>>
>> As per your instructions, I've used the set for <<documents>> <<ranks>>
>> <<visits>> <<results>>
>> that I downloaded from your scratch tree branch.
>>
>> It seems that the main function fails because of other reasons (even
>> before entering the programme logic).
>> I'm trying to break my head on the exact reason ....
>>
>> If you could help me in this direction ... ?
>>
>> Thanks a million in advance,
>> Anirvan
>>
>>
>> Le 23/09/2014 17:22, rmetzger0 [via Apache Flink (Incubator) User Mailing
>> List archive.] a écrit :
>>
>> Hi,
>>
>>  you have to use the "WebLogDataGenerator" found here:
>> https://github.com/apache/incubator-flink/blob/master/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/relational/util/WebLogDataGenerator.java
>>
>>  It accepts two arguments, the number of documents and visits.
>> The generated files are located in /tmp/documents /tmp/ranks and
>> /tmp/visits.
>> I've generated some sample data for you, located here:
>> https://github.com/rmetzger/scratch/tree/weblogdataexample/weblog
>>
>>
>>  Best,
>> Robert
>>
>>
>> On Tue, Sep 23, 2014 at 4:05 PM, nirvanesque [via Apache Flink
>> (Incubator) User Mailing List archive.] <[hidden email]
>> <http://user/SendEmail.jtp?type=node&node=100&i=0>> wrote:
>>
>>> Hello Robert,
>>>
>>> Thanks as usual for all your help with the information.
>>>
>>> I'm trying in vain to create the different input files from the program
>>> source code but running into difficulties.
>>>
>>> Could you (or anyone else) please post here samples of the 4 inputs that
>>> are required to run this program ?
>>>
>>> Thanks in advance,
>>> Anirvan
>>>
>>> Le 09/09/2014 23:54, rmetzger0 [via Apache Flink (Incubator) User
>>> Mailing List archive.] a écrit :
>>>
>>> Hi Anirvan,
>>>
>>> sorry for the late response. You've posted the question to Nabble, which
>>> is only a mirror of our actual mailing list at [hidden email]
>>> <http://user/SendEmail.jtp?type=node&node=99&i=0>. Sadly, the message
>>> is not automatically posted to the apache list because the apache server is
>>> rejecting the mails from nabble.
>>>  I've already asked and there is no way to change this behavior.
>>> So I actually saw the two messages you posted here by accident.
>>>
>>> Regarding your actual question:
>>> - The command line arguments for the WebLogAnalysis example are:
>>>    "WebLogAnalysis <documents path> <ranks path> <visits path>
<result
>>> path>"
>>>
>>> - Regarding the "info -d" command. I think its an artifact from our old
>>> java API. I've filed an issue in JIRA:
>>> https://issues.apache.org/jira/browse/FLINK-1095 Lets see how we
>>> resolve it. Thanks for reporting this!
>>>
>>> You can find the source code of all of our examples in the source
>>> release of Flink (in the flink-examples/flink-java-examples project. You
>>> can also access the source (and hence the examples) through GitHub:
>>> https://github.com/apache/incubator-flink/blob/master/flink-examples/flink-java-examples/src/main/java/org/apache/flink/example/java/relational/WebLogAnalysis.java.
>>>
>>>
>>> To build the examples, you can run: "mvn clean package -DskipTests" in
>>> the "flink-examples/flink-java-examples" directory. This will re-build
>>> them.
>>>
>>> If you don't want to import the whole Flink project just for playing
>>> around with the examples, you can also create an empty maven project. This
>>> script:
>>> curl
>>> https://raw.githubusercontent.com/apache/incubator-flink/master/flink-quickstart/quickstart.sh
|
>>> bash
>>>
>>> will automatically set everything up for you. Just import the
>>> "quickstart" project into Eclipse or IntelliJ. It will download all
>>> dependencies and package everything correctly. If you want to use an
>>> example there, just copy the Java file into the "quickstart" project.
>>>
>>> The examples are indeed a very good way to learn how to write Flink
>>> jobs.
>>>
>>> Please continue asking if you have further questions!
>>>
>>> Best,
>>> Robert
>>>
>>>
>>>  ------------------------------
>>>  If you reply to this email, your message will be added to the
>>> discussion below:
>>>
>>> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p67.html
>>>  To unsubscribe from Looking for instructions & source for
>>> flink-java-examples-0.6-incubating-WebLogAnalysis.jar, click here.
>>> NAML
>>> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>> ------------------------------
>>>  If you reply to this email, your message will be added to the
>>> discussion below:
>>>
>>> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p99.html
>>>  To unsubscribe from Looking for instructions & source for
>>> flink-java-examples-0.6-incubating-WebLogAnalysis.jar, click here.
>>> NAML
>>> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>
>>
>>
>> ------------------------------
>>  If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p100.html
>>  To unsubscribe from Looking for instructions & source for
>> flink-java-examples-0.6-incubating-WebLogAnalysis.jar, click here.
>> NAML
>> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>> ------------------------------
>>  If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p102.html
>>  To unsubscribe from Looking for instructions & source for
>> flink-java-examples-0.6-incubating-WebLogAnalysis.jar, click here.
>> NAML
>> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Looking for instructions & source for
> flink-java-examples-0.6-incubating-WebLogAnalysis.jar
> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p103.html>
>
> Sent from the Apache Flink (Incubator) User Mailing List archive. mailing
> list archive
> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/>
> at Nabble.com.
>

Mime
View raw message