Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
References: <CAOhmDzc5AQu8t=c6o_msOfhJ8MLHhMpqZh_+r7_+CdQWa9Vgew@mail.gmail.com>
 <CAAOnQ7uKe+af-khcv5bH=224mBDaWLrdc1uSZrtYR2mzwfVYow@mail.gmail.com>
In-Reply-To: <CAAOnQ7uKe+af-khcv5bH=224mBDaWLrdc1uSZrtYR2mzwfVYow@mail.gmail.com>
From: Nicholas Chammas <nicholas.chammas@gmail.com>
Date: Fri, 29 Jul 2016 20:13:08 +0000
Message-ID: <CAOhmDzcTBFaki97__xw-2dEHYz-y2ks1Br4zXGKKsizom_JKDw@mail.gmail.com>
Subject: Re: Clarifying that spark-x.x.x-bin-hadoopx.x.tgz doesn't include
 Hadoop itself
To: Marcelo Vanzin <vanzin@cloudera.com>
Cc: Spark dev list <dev@spark.apache.org>
Content-Type: multipart/alternative; boundary=001a113ba24899dee70538cbe175
archived-at: Fri, 29 Jul 2016 20:13:25 -0000

--001a113ba24899dee70538cbe175
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hmm, perhaps I'm the one who's confused. =F0=9F=A4=94

I thought the person in the linked discussion expected Hadoop itself (i.e.
the full application, not just the jars) to somehow be included, but
rereading the discussion I may have just misinterpreted them.

The Hadoop jars packaged with Spark just allow Spark to interact with
Hadoop, or allow it to use the Hadoop API for interacting with systems like
S3, right? If you want HDFS, MapReduce, etc. you're obviously not getting
that in those Spark packages.

Maybe this was already clear to those users and I just injected some
confusion into the discussion.

Nick

On Fri, Jul 29, 2016 at 4:06 PM Marcelo Vanzin <vanzin@cloudera.com> wrote:

> Why do you say Hadoop is not included?
>
> The Hadoop jars are there in the tarball, and match the advertised
> version. There is (or at least there was in 1.x) a version called
> "without-hadoop" which did not include any Hadoop jars.
>
> On Fri, Jul 29, 2016 at 12:56 PM, Nicholas Chammas
> <nicholas.chammas@gmail.com> wrote:
> > I had an interaction on my project today that suggested some people may
> be
> > confused about what the packages available on the downloads page are
> > actually for.
> >
> > Specifically, the various -hadoopx.x.tgz packages suggest that Hadoop
> itself
> > is actually included in the package. I=E2=80=99m not 100% sure myself h=
onestly,
> but
> > as I explained in my comment linked above, I believe the -hadoopx.x.tgz
> just
> > indicates the version of Hadoop that Spark was built against.
> >
> > Does it make sense to add a brief note to the downloads page explaining
> > this?
> >
> > I am assuming it would be too disruptive to change the package names to
> > something more descriptive like -built-against-hadoopx.x.tgz.
> >
> > Nick
>
>
>
> --
> Marcelo
>

--001a113ba24899dee70538cbe175
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hmm, perhaps I&#39;m the one who&#39;s confused. =F0=9F=A4=
=94<div><br></div><div>I thought the person in the linked discussion expect=
ed Hadoop itself (i.e. the full application, not just the jars) to somehow =
be included, but rereading the discussion I may have just misinterpreted th=
em.</div><div><br></div><div>The Hadoop jars packaged with Spark just allow=
 Spark to interact with Hadoop, or allow it to use the Hadoop API for inter=
acting with systems like S3, right?=C2=A0<span style=3D"line-height:1.5">If=
 you want HDFS, MapReduce, etc. you&#39;re obviously not getting that in th=
ose Spark packages.</span></div><div><br></div><div>Maybe this was already =
clear to those users and I just injected some confusion into the discussion=
.</div><div><br></div><div>Nick</div></div><br><div class=3D"gmail_quote"><=
div dir=3D"ltr">On Fri, Jul 29, 2016 at 4:06 PM Marcelo Vanzin &lt;<a href=
=3D"mailto:vanzin@cloudera.com">vanzin@cloudera.com</a>&gt; wrote:<br></div=
><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1=
px #ccc solid;padding-left:1ex">Why do you say Hadoop is not included?<br>
<br>
The Hadoop jars are there in the tarball, and match the advertised<br>
version. There is (or at least there was in 1.x) a version called<br>
&quot;without-hadoop&quot; which did not include any Hadoop jars.<br>
<br>
On Fri, Jul 29, 2016 at 12:56 PM, Nicholas Chammas<br>
&lt;<a href=3D"mailto:nicholas.chammas@gmail.com" target=3D"_blank">nichola=
s.chammas@gmail.com</a>&gt; wrote:<br>
&gt; I had an interaction on my project today that suggested some people ma=
y be<br>
&gt; confused about what the packages available on the downloads page are<b=
r>
&gt; actually for.<br>
&gt;<br>
&gt; Specifically, the various -hadoopx.x.tgz packages suggest that Hadoop =
itself<br>
&gt; is actually included in the package. I=E2=80=99m not 100% sure myself =
honestly, but<br>
&gt; as I explained in my comment linked above, I believe the -hadoopx.x.tg=
z just<br>
&gt; indicates the version of Hadoop that Spark was built against.<br>
&gt;<br>
&gt; Does it make sense to add a brief note to the downloads page explainin=
g<br>
&gt; this?<br>
&gt;<br>
&gt; I am assuming it would be too disruptive to change the package names t=
o<br>
&gt; something more descriptive like -built-against-hadoopx.x.tgz.<br>
&gt;<br>
&gt; Nick<br>
<br>
<br>
<br>
--<br>
Marcelo<br>
</blockquote></div>

--001a113ba24899dee70538cbe175--