beam-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Punit Naik <naik.puni...@gmail.com>
Subject Re: SparkPipelineRunner ERROR
Date Wed, 25 May 2016 09:36:36 GMT
Hi Everyone

I was able to run the example with Spark runner on files present in /tmp
directory
. What if I want to use the files from HDFS?

On Wed, May 25, 2016 at 2:04 PM, Punit Naik <naik.punit44@gmail.com> wrote:

> Hi Ismael
>
> I am using maven 3.3.9
>
> On Wed, May 25, 2016 at 1:31 PM, Ismaël Mejía <iemejia@gmail.com> wrote:
>
>> Hi Punit,
>>
>> Can you confirm the version of Maven are you using ?
>>
>> I ask this because I had some issues building the project with the native
>> version of maven that came with debian/ubuntu, but since I moved to the
>> 'real' one I have no problmems. And I just did git pull from master and
>> everything seems to work ok.
>>
>> Ismaël
>>
>>
>> On Wed, May 25, 2016 at 10:00 AM, Amit Sela <amitsela33@gmail.com> wrote:
>>
>>> I'll take a look first chance I get and get back to you.
>>>
>>> On Wed, May 25, 2016, 10:57 Punit Naik <naik.punit44@gmail.com> wrote:
>>>
>>>> Hi Amit
>>>>
>>>> Thanks, it ran but got stuck at:
>>>>
>>>> [INFO] --- exec-maven-plugin:1.4.0:exec (default-cli) @ spark ---
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.beam.sdk.options.PipelineOptionsFactory).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>> On Wed, May 25, 2016 at 1:17 PM, Amit Sela <amitsela33@gmail.com>
>>>> wrote:
>>>>
>>>>> I'm not near a computer but are you running from the parent project
>>>>> dir or the runner ? You need to run from the runner dir.
>>>>>
>>>>> On Wed, May 25, 2016, 10:44 Punit Naik <naik.punit44@gmail.com>
wrote:
>>>>>
>>>>>> Hi Amit
>>>>>>
>>>>>> Just tried that and still got an error:
>>>>>>
>>>>>> [ERROR] Failed to execute goal
>>>>>> org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (default-cli) on project
>>>>>> parent: The parameters 'executable' for goal
>>>>>> org.codehaus.mojo:exec-maven-plugin:1.4.0:exec are missing or invalid
->
>>>>>> [Help 1]
>>>>>>
>>>>>> On Wed, May 25, 2016 at 1:01 PM, Amit Sela <amitsela33@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Would you try running with mvn exec:exec ?
>>>>>>> See here:
>>>>>>> https://github.com/apache/incubator-beam/blob/master/runners/spark/README.md
>>>>>>>
>>>>>>> On Wed, May 25, 2016, 10:14 Punit Naik <naik.punit44@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Everyone
>>>>>>>>
>>>>>>>> I compiled beam project with the command "mvn clean install
>>>>>>>> -DskipTests" and then ran the WordCount example with the
following command:
>>>>>>>>
>>>>>>>> mvn compile exec:java -pl examples/java
>>>>>>>> -Dexec.mainClass=org.apache.beam.examples.WordCount
>>>>>>>> -Dexec.args="--runner=SparkPipelineRunner --sparkMaster=local
>>>>>>>> --inputFile=/in --output=/beam-out"
>>>>>>>>
>>>>>>>> But it gave me an error saying:
>>>>>>>>
>>>>>>>> [ERROR] Failed to execute goal
>>>>>>>> org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli)
on project
>>>>>>>> java-examples-all: An exception occured while executing the
Java class.
>>>>>>>> null: InvocationTargetException: Unknown 'runner' specified
>>>>>>>> 'SparkPipelineRunner', supported pipeline runners
>>>>>>>> [BlockingDataflowPipelineRunner, DataflowPipelineRunner,
>>>>>>>> DirectPipelineRunner] -> [Help 1]
>>>>>>>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed
to
>>>>>>>> execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java
(default-cli)
>>>>>>>> on project java-examples-all: An exception occured while
executing the Java
>>>>>>>> class. null
>>>>>>>>     at
>>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
>>>>>>>>     at
>>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>>>>>>>>     at
>>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>>>>>>>>     at
>>>>>>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
>>>>>>>>     at
>>>>>>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
>>>>>>>>     at
>>>>>>>> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>>>>>>>>     at
>>>>>>>> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
>>>>>>>>     at
>>>>>>>> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>>>>>>>>     at
>>>>>>>> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>>>>>>>>     at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>>>>>>>>     at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
>>>>>>>>     at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
>>>>>>>>     at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
>>>>>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
>>>>>>>>     at
>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>>>     at
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>     at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>>>>     at
>>>>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>>>>>>>>     at
>>>>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>>>>>>>>     at
>>>>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>>>>>>>>     at
>>>>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
>>>>>>>> Caused by: org.apache.maven.plugin.MojoExecutionException:
An
>>>>>>>> exception occured while executing the Java class. null
>>>>>>>>     at
>>>>>>>> org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:345)
>>>>>>>>     at
>>>>>>>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
>>>>>>>>     at
>>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
>>>>>>>>     ... 20 more
>>>>>>>> Caused by: java.lang.reflect.InvocationTargetException
>>>>>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
>>>>>>>>     at
>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>>>     at
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>     at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>>>>     at
>>>>>>>> org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
>>>>>>>>     at java.lang.Thread.run(Thread.java:745)
>>>>>>>> Caused by: java.lang.IllegalArgumentException: Unknown 'runner'
>>>>>>>> specified 'SparkPipelineRunner', supported pipeline runners
>>>>>>>> [BlockingDataflowPipelineRunner, DataflowPipelineRunner,
>>>>>>>> DirectPipelineRunner]
>>>>>>>>     at
>>>>>>>> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1434)
>>>>>>>>     at
>>>>>>>> org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:107)
>>>>>>>>     at
>>>>>>>> org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:292)
>>>>>>>>     at org.apache.beam.examples.WordCount.main(WordCount.java:192)
>>>>>>>>     ... 6 more
>>>>>>>> Caused by: java.lang.ClassNotFoundException: SparkPipelineRunner
>>>>>>>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>>>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>>>>     at java.lang.Class.forName0(Native Method)
>>>>>>>>     at java.lang.Class.forName(Class.java:264)
>>>>>>>>     at
>>>>>>>> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1421)
>>>>>>>>     ... 9 more
>>>>>>>>
>>>>>>>> What is the problem?
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thank You
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Punit Naik
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thank You
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Punit Naik
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Thank You
>>>>
>>>> Regards
>>>>
>>>> Punit Naik
>>>>
>>>
>>
>
>
> --
> Thank You
>
> Regards
>
> Punit Naik
>



-- 
Thank You

Regards

Punit Naik

Mime
View raw message