spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Ahuja <aahuj...@gmail.com>
Subject Re: unable to bring up cluster with ec2 script
Date Tue, 07 Jul 2015 20:42:57 GMT
Sorry, I can't help with this issue, but if you are interested in a simple
way to launch a Spark cluster on Amazon, Spark is now offered as an
application in Amazon EMR.  With this you can have a full cluster with a
few clicks:

https://aws.amazon.com/blogs/aws/new-apache-spark-on-amazon-emr/

- Arun

On Tue, Jul 7, 2015 at 4:34 PM, Pagliari, Roberto <rpagliari@appcomsci.com>
wrote:

>
>
>
>
> I'm following the tutorial about Apache Spark on EC2. The output is the
> following:
>
>
>
>
>
>     $ ./spark-ec2 -i ../spark.pem -k spark --copy launch spark-training
>
>     Setting up security groups...
>
>     Searching for existing cluster spark-training...
>
>     Latest Spark AMI: ami-19474270
>
>     Launching instances...
>
>     Launched 5 slaves in us-east-1d, regid = r-59a0d4b6
>
>     Launched master in us-east-1d, regid = r-9ba2d674
>
>     Waiting for instances to start up...
>
>     Waiting 120 more seconds...
>
>     Copying SSH key ../spark.pem to master...
>
>     ssh: connect to host ec2-54-152-15-165.compute-1.amazonaws.com port
> 22: Connection refused
>
>     Error connecting to host Command 'ssh -t -o StrictHostKeyChecking=no
> -i ../spark.pem root@ec2-54-152-15-165.compute-1.amazonaws.com 'mkdir -p
> ~/.ssh'' returned non-zero exit status 255, sleeping 30
>
>     ssh: connect to host ec2-54-152-15-165.compute-1.amazonaws.com port
> 22: Connection refused
>
>     Error connecting to host Command 'ssh -t -o StrictHostKeyChecking=no
> -i ../spark.pem root@ec2-54-152-15-165.compute-1.amazonaws.com 'mkdir -p
> ~/.ssh'' returned non-zero exit status 255, sleeping 30
>
>     ssh: Could not resolve hostname
> ec2-54-152-15-165.compute-1.amazonaws.com: Name or service not known
>
>     Error connecting to host Command 'ssh -t -o StrictHostKeyChecking=no
> -i ../spark.pem root@ec2-54-152-15-165.compute-1.amazonaws.com 'mkdir -p
> ~/.ssh'' returned non-zero exit status 255, sleeping 30
>
>     ssh: connect to host ec2-54-152-15-165.compute-1.amazonaws.com port
> 22: Connection refused
>
>    Traceback (most recent call last):
>
>       File "./spark_ec2.py", line 925, in <module>
>
>         main()
>
>       File "./spark_ec2.py", line 766, in main
>
>         setup_cluster(conn, master_nodes, slave_nodes, zoo_nodes, opts,
> True)
>
>       File "./spark_ec2.py", line 406, in setup_cluster
>
>         ssh(master, opts, 'mkdir -p ~/.ssh')
>
>       File "./spark_ec2.py", line 712, in ssh
>
>         raise e
>
>     subprocess.CalledProcessError: Command 'ssh -t -o
> StrictHostKeyChecking=no -i ../spark.pem
> root@ec2-54-152-15-165.compute-1.amazonaws.com 'mkdir -p ~/.ssh''
> returned non-zero exit status 255
>
>
>
>
>
> However, I can see the six instances created on my EC2 console, and I
> could even get the name of the master. I'm not sure how to fix the ssh
> issue (my region is US EST).
>
>
>

Mime
View raw message