spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohith Parameshwara <rparameshw...@couponsinc.com>
Subject Unable to run applications on spark in standalone cluster mode
Date Mon, 19 Oct 2015 12:05:51 GMT
Hi all,
                I am doing some experiments on spark standalone cluster setup and I am facing
the following issue:
I have a 4 node cluster setup. As per http://spark.apache.org/docs/latest/spark-standalone.html#starting-a-cluster-manually
I tried to start the cluster with the scripts but, the slaves did not start and gave permission
denied error....My conf/slaves had a list of IP addresses of the slaves... But Then I was
able to start the worker nodes by going to the respective slave machine and running the start-slave.sh
script with the master IP as parameter.... The webui showed the all the 3 worker nodes running
as :
[cid:image001.png@01D10A93.B59D7300]

The application when submitted to the cluster will show in all the work nodes in the webui....
But in the command line where is spark-submit command was run, it gives this error periodically
and continuously....
ERROR NettyTransport: failed to bind to /172.28.161.138:7077, shutting down Netty transport
15/10/19 11:59:38 WARN Utils: Service 'sparkDriver' could not bind on port 7077. Attempting
port 7078.
15/10/19 11:59:38 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@localhost:7077:
akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077....

My conf/spark-env.sh has:

SPARK_MASTER_IP=172.28.161.138
SPARK_MASTER_PORT=7077
SPARK_MASTER_WEBUI_PORT=8080
SPARK_WORKER_WEBUI_PORT=8081
SPARK_WORKER_INSTANCES=1
And I have also put this in all the slave nodes too....

The applications are running fine in -master local mode but in -master spark://masterip:7077,
it is not working....
Any type of help would be appreciated..... Thanks in advance

Mime
View raw message