spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Giridhar Maddukuri <giridhar.madduk...@gmail.com>
Subject Re: SPARK SQL Error
Date Thu, 15 Oct 2015 12:56:23 GMT
Hi Dilip,

I tried this option also spark-submit --master yarn --class
org.spark.apache.CsvDataSource /home/cloudera/Desktop/TestMain.jar  --files
hdfs://quickstart.cloudera:8020/people_csv & getting similar error

Exception in thread "main" org.apache.spark.SparkException: Could not parse
Master URL: '--files'
 at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2244)
 at org.apache.spark.SparkContext.<init>(SparkContext.scala:361)
 at org.apache.spark.SparkContext.<init>(SparkContext.scala:154)
 at org.spark.apache.CsvDataSource$.main(CsvDataSource.scala:10)
 at org.spark.apache.CsvDataSource.main(CsvDataSource.scala)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
 at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
 at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
 at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Thanks & Regards,
Giri.


On Thu, Oct 15, 2015 at 5:43 PM, Dilip Biswal <dbiswal@us.ibm.com> wrote:

> Hi Giri,
>
> You are perhaps  missing the "--files" option before the supplied hdfs
> file name ?
>
> spark-submit --master yarn --class org.spark.apache.CsvDataSource
> /home/cloudera/Desktop/TestMain.jar  *--files*
> hdfs://quickstart.cloudera:8020/people_csv
>
> Please refer to Ritchard's comments on why the --files option may be
> redundant in
> your case.
>
> Regards,
> Dilip Biswal
> Tel: 408-463-4980
> dbiswal@us.ibm.com
>
>
>
> From:        Giri <giridhar.maddukuri@gmail.com>
> To:        user@spark.apache.org
> Date:        10/15/2015 02:44 AM
> Subject:        Re: SPARK SQL Error
> ------------------------------
>
>
>
> Hi Ritchard,
>
> Thank you so much  again for your input.This time I ran the command in the
> below way
> spark-submit --master yarn --class org.spark.apache.CsvDataSource
> /home/cloudera/Desktop/TestMain.jar
> hdfs://quickstart.cloudera:8020/people_csv
> But I am facing the new error "Could not parse Master URL:
> 'hdfs://quickstart.cloudera:8020/people_csv'"
> file path is correct
>
> hadoop fs -ls hdfs://quickstart.cloudera:8020/people_csv
> -rw-r--r--   1 cloudera supergroup         29 2015-10-10 00:02
> hdfs://quickstart.cloudera:8020/people_csv
>
> Can you help me to fix this new error
>
> 15/10/15 02:24:39 INFO spark.SparkContext: Added JAR
> file:/home/cloudera/Desktop/TestMain.jar at
> http://10.0.2.15:40084/jars/TestMain.jarwith timestamp 1444901079484
> Exception in thread "main" org.apache.spark.SparkException: Could not parse
> Master URL: 'hdfs://quickstart.cloudera:8020/people_csv'
>                 at
>
> org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2244)
>                 at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:361)
>                 at
> org.apache.spark.SparkContext.<init>(SparkContext.scala:154)
>                 at
> org.spark.apache.CsvDataSource$.main(CsvDataSource.scala:10)
>                 at org.spark.apache.CsvDataSource.main(CsvDataSource.scala)
>                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>                 at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>                 at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>                 at java.lang.reflect.Method.invoke(Method.java:606)
>                 at
>
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
>                 at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
>                 at
> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
>                 at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
>                 at
> org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
> Thanks & Regards,
> Giri.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/SPARK-SQL-Error-tp25050p25075.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>
>

Mime
View raw message