Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
From: Matt Goodman <meawoppl@gmail.com>
Date: Sat, 11 Jul 2015 11:07:12 -0700
Message-ID: 
 <CAKjxZ9TJM9sLPvS_-mw0VSQsa-je8NJgGTe5iPt_O5qKULVEAQ@mail.gmail.com>
Subject: Re: Should spark-ec2 get its own repo?
To: dev@spark.apache.org
Content-Type: multipart/alternative; boundary=047d7b163515c5e928051a9d5c42

--047d7b163515c5e928051a9d5c42
Content-Type: text/plain; charset=UTF-8

I wanted to revive the conversation about the spark-ec2 tools, as it seems
to have been lost in the 1.4.1 release voting spree.

I think that splitting it into its own repository is a really good move,
and I would also be happy to help with this transition, as well as help
maintain the resulting repository.  Here is my justification for why we
ought to do this split.

User Facing:

   - The spark-ec2 launcher dosen't use anything in the parent spark
   repository
   - spark-ec2 version is disjoint from the parent repo.  I consider it
   confusing that the spark-ec2 script dosen't launch the version of spark it
   is checked-out with.
   - Someone interested in setting up spark-ec2 with anything but the
   default configuration will have to clone at least 2 repositories at
   present, and probably fork and push changes to 1.
   - spark-ec2 has mismatched dependencies wrt. to spark itself.  This
   includes a confusing shim in the spark-ec2 script to install boto, which
   frankly should just be a dependency of the script

Developer Facing:

   - Support across 2 repos will be worse than across 1.  Its unclear where
   to file issues/PRs, and requires extra communications for even fairly
   trivial stuff.
   - Spark-ec2 also depends on a number binary blobs being in the right
   place, currently the responsibility for these is decentralized, and likely
   prone to various flavors of dumb.
   - The current flow of booting a spark-ec2 cluster is _complicated_ I
   spent the better part of a couple days figuring out how to integrate our
   custom tools into this stack.  This is very hard to fix when commits/PR's
   need to span groups/repositories/buckets-o-binary, I am sure there are
   several other problems that are languishing under similar roadblocks
   - It makes testing possible.  The spark-ec2 script is a great case for
   CI given the number of permutations of launch criteria there are.  I
   suspect AWS would be happy to foot the bill on spark-ec2 testing (probably
   ~20 bucks a month based on some envelope sketches), as it is a piece of
   software that directly impacts other people giving them money.  I have some
   contacts there, and I am pretty sure this would be an easy conversation,
   particularly if the repo directly concerned with ec2.  Think also being
   able to assemble the binary blobs into s3 bucket dedicated to spark-ec2

Any other thoughts/voices appreciated here.  spark-ec2 is a super-power
tool and deserves a fair bit of attention!
--Matthew Goodman

=====================
Check Out My Website: http://craneium.net
Find me on LinkedIn: http://tinyurl.com/d6wlch

--047d7b163515c5e928051a9d5c42
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">I wanted to revive the conversation about the spark-ec2 to=
ols, as it seems to have been lost in the 1.4.1 release voting spree.<div><=
br></div><div>I think that splitting it into its own repository is a really=
 good move, and I would also be happy to help with this transition, as well=
 as help maintain the resulting repository.=C2=A0 Here is my justification =
for why we ought to do this split.</div><div><br></div><div>User Facing:</d=
iv><div><ul><li>The spark-ec2 launcher dosen&#39;t use anything in the pare=
nt spark repository</li><li>spark-ec2 version is disjoint from the parent r=
epo.=C2=A0 I consider it confusing that the spark-ec2 script dosen&#39;t la=
unch the version of spark it is checked-out with.</li><li>Someone intereste=
d in setting up spark-ec2 with anything but the default configuration will =
have to clone at least 2 repositories at present, and probably fork and pus=
h changes to 1.</li><li>spark-ec2 has mismatched dependencies wrt. to spark=
 itself.=C2=A0 This includes a confusing shim in the spark-ec2 script to in=
stall boto, which frankly should just be a dependency of the script</li></u=
l>Developer Facing:<br><ul><li>Support across 2 repos will be worse than ac=
ross 1.=C2=A0 Its unclear where to file issues/PRs, and requires extra comm=
unications for even fairly trivial stuff.</li><li>Spark-ec2 also depends on=
 a number binary blobs being in the right place, currently the responsibili=
ty for these is decentralized, and likely prone to various flavors of dumb.=
</li><li>The current flow of booting a spark-ec2 cluster is _complicated_ I=
 spent the better part of a couple days figuring out how to integrate our c=
ustom tools into this stack.=C2=A0 This is very hard to fix when commits/PR=
&#39;s need to span groups/repositories/buckets-o-binary, I am sure there a=
re several other problems that are languishing under similar roadblocks</li=
><li>It makes testing possible.=C2=A0 The spark-ec2 script is a great case =
for CI given the number of permutations of launch criteria there are.=C2=A0=
 I suspect AWS would be happy to foot the bill on spark-ec2 testing (probab=
ly ~20 bucks a month based on some envelope sketches), as it is a piece of =
software that directly impacts other people giving them money.=C2=A0 I have=
 some contacts there, and I am pretty sure this would be an easy conversati=
on, particularly if the repo directly concerned with ec2.=C2=A0 Think also =
being able to assemble the binary blobs into s3 bucket dedicated to spark-e=
c2 =C2=A0</li></ul><div><div class=3D"gmail_signature">Any other thoughts/v=
oices appreciated here. =C2=A0spark-ec2 is a super-power tool and deserves =
a fair bit of attention!</div><div class=3D"gmail_signature">--Matthew Good=
man<br><br>=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D<=
br>Check Out My Website: <a href=3D"http://craneium.net" target=3D"_blank">=
http://craneium.net</a><br>Find me on LinkedIn: <a href=3D"http://tinyurl.c=
om/d6wlch" target=3D"_blank">http://tinyurl.com/d6wlch</a><br></div></div>
</div></div>

--047d7b163515c5e928051a9d5c42--