flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rmetzger0 <rmetz...@apache.org>
Subject Re: Looking for instructions & source for flink-java-examples-0.6-incubating-WebLogAnalysis.jar
Date Wed, 24 Sep 2014 15:48:29 GMT
Hi,

this looks like a minor issue in your flink configuration.
Can you have a look into your config (in the conf/flink-conf.yaml file) and
tell me the value of the "parallelization.degree.default" property?
If it is set to 0 or a negative value, thats the source of the issue. Set
it to something higher (I recommend numOfNodes*numOfCoresPerNode).

But I have to admit that our error reporting here is not very helpful,
we'll improve that...


Cheers,
Robert


On Wed, Sep 24, 2014 at 4:23 PM, nirvanesque [via Apache Flink (Incubator)
User Mailing List archive.] <ml-node+s2336050n102h7@n4.nabble.com> wrote:

> Hello robert,
>
> Apologies for getting back to you on this one :-(
> Es tut mir wirklich Leid!
>
> I've been trying today several times in vain this programme, and following
> is the error output :
>
> [QUOTE]
> abasu@edel-4:~$ ./flink/bin/flink run -v
> flink/examples/flink-java-examples-0.6-incubating-WebLogAnalysis.jar
> file:///home/abasu/examples/Weblogs/documents
> file:///home/abasu/examples/Weblogs/ranks
> file:///home/abasu/examples/Weblogs/visits
> file:///home/abasu/examples/Weblogs/results
> Error: The main method caused an error.
> org.apache.flink.client.program.ProgramInvocationException: The main
> method caused an error.
>         at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:404)
>         at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:307)
>         at org.apache.flink.client.program.Client.run(Client.java:244)
>         at
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:332)
>         at org.apache.flink.client.CliFrontend.run(CliFrontend.java:319)
>         at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:930)
>         at org.apache.flink.client.CliFrontend.main(CliFrontend.java:954)
> Caused by: java.lang.IllegalArgumentException
>         at
> org.apache.flink.compiler.dag.OptimizerNode.setDegreeOfParallelism(OptimizerNode.java:414)
>         at
> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.preVisit(PactCompiler.java:772)
>         at
> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.preVisit(PactCompiler.java:616)
>         at
> org.apache.flink.api.common.operators.base.GenericDataSinkBase.accept(GenericDataSinkBase.java:287)
>         at org.apache.flink.api.common.Plan.accept(Plan.java:281)
>         at
> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:511)
>         at
> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:460)
>         at
> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:196)
>         at
> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:209)
>         at org.apache.flink.client.program.Client.run(Client.java:285)
>         at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:54)
>         at
> org.apache.flink.example.java.relational.WebLogAnalysis.main(WebLogAnalysis.java:148)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:389)
>         ... 6 more
> abasu@edel-4:~$
>
> [/QUOTE]
>
>
> I'm using Flink version 0.6 on a cluster of 10 nodes.
> Job launched from Job manager node.
>
> As per your instructions, I've used the set for <<documents>> <<ranks>>
> <<visits>> <<results>>
> that I downloaded from your scratch tree branch.
>
> It seems that the main function fails because of other reasons (even
> before entering the programme logic).
> I'm trying to break my head on the exact reason ....
>
> If you could help me in this direction ... ?
>
> Thanks a million in advance,
> Anirvan
>
>
> Le 23/09/2014 17:22, rmetzger0 [via Apache Flink (Incubator) User Mailing
> List archive.] a écrit :
>
> Hi,
>
>  you have to use the "WebLogDataGenerator" found here:
> https://github.com/apache/incubator-flink/blob/master/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/relational/util/WebLogDataGenerator.java
>
>  It accepts two arguments, the number of documents and visits.
> The generated files are located in /tmp/documents /tmp/ranks and
> /tmp/visits.
> I've generated some sample data for you, located here:
> https://github.com/rmetzger/scratch/tree/weblogdataexample/weblog
>
>
>  Best,
> Robert
>
>
> On Tue, Sep 23, 2014 at 4:05 PM, nirvanesque [via Apache Flink (Incubator)
> User Mailing List archive.] <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=100&i=0>> wrote:
>
>> Hello Robert,
>>
>> Thanks as usual for all your help with the information.
>>
>> I'm trying in vain to create the different input files from the program
>> source code but running into difficulties.
>>
>> Could you (or anyone else) please post here samples of the 4 inputs that
>> are required to run this program ?
>>
>> Thanks in advance,
>> Anirvan
>>
>> Le 09/09/2014 23:54, rmetzger0 [via Apache Flink (Incubator) User Mailing
>> List archive.] a écrit :
>>
>> Hi Anirvan,
>>
>> sorry for the late response. You've posted the question to Nabble, which
>> is only a mirror of our actual mailing list at [hidden email]
>> <http://user/SendEmail.jtp?type=node&node=99&i=0>. Sadly, the message
is
>> not automatically posted to the apache list because the apache server is
>> rejecting the mails from nabble.
>>  I've already asked and there is no way to change this behavior.
>> So I actually saw the two messages you posted here by accident.
>>
>> Regarding your actual question:
>> - The command line arguments for the WebLogAnalysis example are:
>>    "WebLogAnalysis <documents path> <ranks path> <visits path>
<result
>> path>"
>>
>> - Regarding the "info -d" command. I think its an artifact from our old
>> java API. I've filed an issue in JIRA:
>> https://issues.apache.org/jira/browse/FLINK-1095 Lets see how we resolve
>> it. Thanks for reporting this!
>>
>> You can find the source code of all of our examples in the source release
>> of Flink (in the flink-examples/flink-java-examples project. You can also
>> access the source (and hence the examples) through GitHub:
>> https://github.com/apache/incubator-flink/blob/master/flink-examples/flink-java-examples/src/main/java/org/apache/flink/example/java/relational/WebLogAnalysis.java.
>>
>>
>> To build the examples, you can run: "mvn clean package -DskipTests" in
>> the "flink-examples/flink-java-examples" directory. This will re-build
>> them.
>>
>> If you don't want to import the whole Flink project just for playing
>> around with the examples, you can also create an empty maven project. This
>> script:
>> curl
>> https://raw.githubusercontent.com/apache/incubator-flink/master/flink-quickstart/quickstart.sh
|
>> bash
>>
>> will automatically set everything up for you. Just import the
>> "quickstart" project into Eclipse or IntelliJ. It will download all
>> dependencies and package everything correctly. If you want to use an
>> example there, just copy the Java file into the "quickstart" project.
>>
>> The examples are indeed a very good way to learn how to write Flink jobs.
>>
>> Please continue asking if you have further questions!
>>
>> Best,
>> Robert
>>
>>
>>  ------------------------------
>>  If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p67.html
>>  To unsubscribe from Looking for instructions & source for
>> flink-java-examples-0.6-incubating-WebLogAnalysis.jar, click here.
>> NAML
>> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>> ------------------------------
>>  If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p99.html
>>  To unsubscribe from Looking for instructions & source for
>> flink-java-examples-0.6-incubating-WebLogAnalysis.jar, click here.
>> NAML
>> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p100.html
>  To unsubscribe from Looking for instructions & source for
> flink-java-examples-0.6-incubating-WebLogAnalysis.jar, click here.
> NAML
> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p102.html
>  To unsubscribe from Looking for instructions & source for
> flink-java-examples-0.6-incubating-WebLogAnalysis.jar, click here
> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=66&code=cm1ldHpnZXJAYXBhY2hlLm9yZ3w2NnwtMTU5OTExMTY4NA==>
> .
> NAML
> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p103.html
Sent from the Apache Flink (Incubator) User Mailing List archive. mailing list archive at
Nabble.com.
Mime
View raw message