spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From camelia <came...@chalmers.se>
Subject What happens in the master or slave launch ?
Date Wed, 07 Oct 2015 14:39:27 GMT
Hello,

I have the following question:

I have two scenarios:
1) in one scenario (if I'm connected on the target node) the master starts
successfully.
Its log contains:

Spark Command: /usr/opt/java/jdk1.7.0_07/jre/bin/java -cp
/home/camelia/spark-1.4.1-bin-hadoop2.6/sbin/../conf/:/home/camelia/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar:/home/camelia/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/home/camelia/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/home/camelia/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar
-Dspark.deploy.defaultCores=2 -Xms512m -Xmx512m -XX:MaxPermSize=256m
org.apache.spark.deploy.master.Master --ip ttitania-6 --port 7077
--webui-port 8080
========================================
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
15/10/07 13:23:55 INFO Master: Registered signal handlers for [TERM, HUP,
INT]
15/10/07 13:23:55 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/10/07 13:23:56 INFO SecurityManager: Changing view acls to: camelia
15/10/07 13:23:56 INFO SecurityManager: Changing modify acls to: camelia
15/10/07 13:23:56 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(camelia); users
with modify permissions: Set(camelia)
15/10/07 13:23:56 INFO Slf4jLogger: Slf4jLogger started
15/10/07 13:23:56 INFO Remoting: Starting remoting
15/10/07 13:23:57 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkMaster@ttitania-6:7077]
15/10/07 13:23:57 INFO Utils: Successfully started service 'sparkMaster' on
port 7077.
15/10/07 13:23:57 INFO Utils: Successfully started service on port 6066.
15/10/07 13:23:57 INFO StandaloneRestServer: Started REST server for
submitting applications on port 6066
15/10/07 13:23:57 INFO Master: Starting Spark master at
spark://ttitania-6:7077
15/10/07 13:23:57 INFO Master: Running Spark version 1.4.1
15/10/07 13:23:57 INFO Utils: Successfully started service 'MasterUI' on
port 8080.
15/10/07 13:23:57 INFO MasterWebUI: Started MasterWebUI at
http://129.11.11.111:8080
15/10/07 13:23:57 INFO Master: I have been elected leader! New state: ALIVE


2) in another scenario (trying to start the spark master remotely, using
qrsh as we are disallowed ssh between cluster nodes), when I try to start
one master, it only partially starts and then shuts down. Its log is:

Spark Command: /usr/opt/java/jdk1.7.0_07/jre/bin/java -cp
/home/camelia/spark-1.4.1-bin-hadoop2.6/sbin/../conf/:/home/camelia/spark-1.4.1-bin-hadoop2.6/lib/spark-assembly-1.4.1-hadoop2.6.0.jar:/home/camelia/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar:/home/camelia/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar:/home/camelia/spark-1.4.1-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar
-Dspark.deploy.defaultCores=2 -Xms512m -Xmx512m -XX:MaxPermSize=256m
org.apache.spark.deploy.master.Master --ip ttitania-6 --port 7077
--webui-port 8080
========================================
Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
15/10/07 12:56:31 INFO Master: Registered signal handlers for [TERM, HUP,
INT]
15/10/07 12:56:32 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/10/07 12:56:32 INFO SecurityManager: Changing view acls to: camelia
15/10/07 12:56:32 INFO SecurityManager: Changing modify acls to: camelia
15/10/07 12:56:32 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(camelia); users
with modify permissions: Set(camelia)



Can you please explain which might be the problem?
Which is the step in Spark's starting the master process that determines
this crash? 

I wait impatiently for your reply.
Many thanks,
Camelia



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/What-happens-in-the-master-or-slave-launch-tp24968.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message