spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Art Peel <found...@gmail.com>
Subject thoughts on spark_ec2.py?
Date Fri, 25 Apr 2014 07:51:36 GMT
I've been setting up Spark cluster on EC2 using the provided
ec2/spark_ec2.py script and am very happy I didn't have to write it from
scratch. Thanks for providing it.

There have been some issues, though, and I have had to make some additions.
 So far, they are all additions of command-line options.  For example, the
original script allows access from anywhere to the various ports.  I've
added an option to specify what net/mask should be allowed to access those
ports.

I've filed a couple of pull requests, but they are not going anywhere.
 Given what I've seen of the traffic on this list, I don't feel that a lot
of the developers are thinking about EC2 setup. I totally agree that it is
not as important as improving the guts of Spark itself; nevertheless, I
feel that being able to run Spark on EC2 smartly and easily is valuable.

So, I have 2 questions for the committers:

1. Is ec2/spark_ec2.py something the committers
a. are not thinking about?
b. are planning to replace?
c. other

2. Should I just start a new project based on ec2/spark_ec2.py but without
all the other stuff and make (and share) my changes there?

Regards,

Art

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message