spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dilip Biswal" <dbis...@us.ibm.com>
Subject Re: SPARK SQL Error
Date Thu, 15 Oct 2015 12:13:01 GMT
Hi Giri,

You are perhaps  missing the "--files" option before the supplied hdfs 
file name ?

spark-submit --master yarn --class org.spark.apache.CsvDataSource
/home/cloudera/Desktop/TestMain.jar  --files 
hdfs://quickstart.cloudera:8020/people_csv

Please refer to Ritchard's comments on why the --files option may be 
redundant in 
your case. 

Regards,
Dilip Biswal
Tel: 408-463-4980
dbiswal@us.ibm.com



From:   Giri <giridhar.maddukuri@gmail.com>
To:     user@spark.apache.org
Date:   10/15/2015 02:44 AM
Subject:        Re: SPARK SQL Error



Hi Ritchard,

Thank you so much  again for your input.This time I ran the command in the
below way
spark-submit --master yarn --class org.spark.apache.CsvDataSource
/home/cloudera/Desktop/TestMain.jar 
hdfs://quickstart.cloudera:8020/people_csv
But I am facing the new error "Could not parse Master URL:
'hdfs://quickstart.cloudera:8020/people_csv'"
file path is correct
 
hadoop fs -ls hdfs://quickstart.cloudera:8020/people_csv
-rw-r--r--   1 cloudera supergroup         29 2015-10-10 00:02
hdfs://quickstart.cloudera:8020/people_csv

Can you help me to fix this new error

15/10/15 02:24:39 INFO spark.SparkContext: Added JAR
file:/home/cloudera/Desktop/TestMain.jar at
http://10.0.2.15:40084/jars/TestMain.jar with timestamp 1444901079484
Exception in thread "main" org.apache.spark.SparkException: Could not 
parse
Master URL: 'hdfs://quickstart.cloudera:8020/people_csv'
                 at
org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2244)
                 at 
org.apache.spark.SparkContext.<init>(SparkContext.scala:361)
                 at 
org.apache.spark.SparkContext.<init>(SparkContext.scala:154)
                 at 
org.spark.apache.CsvDataSource$.main(CsvDataSource.scala:10)
                 at 
org.spark.apache.CsvDataSource.main(CsvDataSource.scala)
                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
                 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                 at java.lang.reflect.Method.invoke(Method.java:606)
                 at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
                 at 
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
                 at 
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
                 at 
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
                 at 
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


Thanks & Regards,
Giri.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/SPARK-SQL-Error-tp25050p25075.html

Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org





Mime
View raw message