spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Risberg <trisb...@pivotal.io>
Subject How to set JAVA_HOME with --deploy-mode cluster
Date Mon, 27 Oct 2014 14:26:34 GMT
Hi,

I'm trying to run the SparkPi example using `--deploy-mode cluster`. Not
having much luck. Looks like the JAVA_HOME on my Mac is used for launching
the app on the Spark cluster running on CentOS. How can I set the JAVA_HOME
or better yet, how can I configure Spark to use the JAVA_HOME configured
for the cluster?

Here is the sysout from launching the job:

$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi
--deploy-mode cluster --master spark://borneo:7077
hdfs://borneo:8020/xd/spark/spark-examples_2.10-1.2.0-SNAPSHOT.jar 100
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
14/10/27 10:19:13 INFO SecurityManager: Changing view acls to: trisberg,
14/10/27 10:19:13 INFO SecurityManager: Changing modify acls to: trisberg,
14/10/27 10:19:13 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(trisberg, );
users with modify permissions: Set(trisberg, )
14/10/27 10:19:13 INFO Slf4jLogger: Slf4jLogger started
14/10/27 10:19:14 INFO Utils: Successfully started service 'driverClient'
on port 49999.
Sending launch command to spark://borneo:7077
Driver successfully submitted as driver-20141027091913-0007
... waiting before polling master for driver state
... polling master for driver state
State of driver-20141027091913-0007 is ERROR
Exception from cluster was: java.io.IOException: Cannot run program
"/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/bin/java"
(in directory
"/home/hadoop/spark-1.1.0-bin-hadoop2.4/work/driver-20141027091913-0007"):
error=2, No such file or directory
java.io.IOException: Cannot run program
"/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/bin/java"
(in directory
"/home/hadoop/spark-1.1.0-bin-hadoop2.4/work/driver-20141027091913-0007"):
error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at
org.apache.spark.deploy.worker.ProcessBuilderLike$$anon$4.start(DriverRunner.scala:238)
at
org.apache.spark.deploy.worker.DriverRunner.runCommandWithRetry(DriverRunner.scala:200)
at org.apache.spark.deploy.worker.DriverRunner.org
$apache$spark$deploy$worker$DriverRunner$$launchDriver(DriverRunner.scala:183)
at
org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:88)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
... 4 more

Thanks,
Thomas

Mime
View raw message