spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Malaska <ted.mala...@cloudera.com>
Subject Re: Spark-Submit issues
Date Wed, 12 Nov 2014 11:23:23 GMT
Hey this is Ted

Are you using Shade when you build your jar and are you using the bigger
jar?  Looks like classes are not included in you jar.

On Wed, Nov 12, 2014 at 2:09 AM, Jeniba Johnson <
Jeniba.Johnson@lntinfotech.com> wrote:

> Hi Hari,
>
> Now Iam trying out the same FlumeEventCount example running with
> spark-submit Instead of run example. The steps I followed is that I have
> exported the JavaFlumeEventCount.java into jar.
>
> The command used is
> ./bin/spark-submit --jars lib/spark-examples-1.1.0-hadoop1.0.4.jar
> --master local --class org.JavaFlumeEventCount  bin/flumeeventcnt2.jar
> localhost 2323
>
> The output is
> 14/11/12 17:55:02 INFO scheduler.ReceiverTracker: Stream 0 received 1
> blocks
> 14/11/12 17:55:02 INFO scheduler.JobScheduler: Added jobs for time
> 1415795102000
>
> If I use this command
>  ./bin/spark-submit --master local --class org.JavaFlumeEventCount
> bin/flumeeventcnt2.jar  localhost 2323
>
> Then I get an error
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/spark/examples/streaming/StreamingExamples
>         at org.JavaFlumeEventCount.main(JavaFlumeEventCount.java:22)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.examples.streaming.StreamingExamples
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
>         ... 8 more
>
>
> I Just wanted to ask is  that it is able to  find spark-assembly.jar but
> why not spark-example.jar.
> The next doubt is  while running FlumeEventCount example through runexample
>
> I get an output as
> Received 4 flume events.
>
> 14/11/12 18:30:14 INFO scheduler.JobScheduler: Finished job streaming job
> 1415797214000 ms.0 from job set of time 1415797214000 ms
> 14/11/12 18:30:14 INFO rdd.MappedRDD: Removing RDD 70 from persistence list
>
> But If I run the same program through Spark-Submit
>
> I get an output as
> 14/11/12 17:55:02 INFO scheduler.ReceiverTracker: Stream 0 received 1
> blocks
> 14/11/12 17:55:02 INFO scheduler.JobScheduler: Added jobs for time
> 1415795102000
>
> So I need a clarification, since in the program the printing statement is
> written as " Received n flume events." So how come Iam able to see as
> "Stream 0 received n blocks".
> And what is the difference of running the program through spark-submit and
> run-example.
>
> Awaiting for your kind reply
>
> Regards,
> Jeniba Johnson
>
>
> ________________________________
> The contents of this e-mail and any attachment(s) may contain confidential
> or privileged information for the intended recipient(s). Unintended
> recipients are prohibited from taking action on the basis of information in
> this e-mail and using or disseminating the information, and must notify the
> sender and delete it from their system. L&T Infotech will not accept
> responsibility or liability for the accuracy or completeness of, or the
> presence of any virus or disabling code in this e-mail"
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message