spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicholas Chammas <nicholas.cham...@gmail.com>
Subject Downloading Hadoop from s3://spark-related-packages/
Date Sun, 01 Nov 2015 03:17:52 GMT
https://s3.amazonaws.com/spark-related-packages/

spark-ec2 uses this bucket to download and install HDFS on clusters. Is it
owned by the Spark project or by the AMPLab?

Anyway, it looks like the latest Hadoop install available on there is
Hadoop 2.4.0.

Are there plans to add newer versions of Hadoop for use by spark-ec2 and
similar tools, or should we just be getting that stuff via an Apache mirror
<http://hadoop.apache.org/releases.html>? The latest version is 2.7.1, by
the way.

The problem with the Apache mirrors, if I am not mistaken, is that you
cannot use a single URL that automatically redirects you to a working
mirror to download Hadoop. You have to pick a specific mirror and pray it
doesn't disappear tomorrow.

Nick

Mime
View raw message