hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "AmazonS3" by SteveLoughran
Date Fri, 06 Mar 2015 18:17:06 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The "AmazonS3" page has been changed by SteveLoughran:
https://wiki.apache.org/hadoop/AmazonS3?action=diff&rev1=17&rev2=18

Comment:
link to the HADOOP-11571 JIRA of 2.7+ fixes to s3a

  Note that by using S3 as an input you lose the data locality optimization, which may be
significant. The general best practise is to copy in data using `distcp` at the start of a
workflow, then copy it out at the end, using the transient HDFS in between.
  
  = History =
-  * The S3 block filesystem was introduced in Hadoop 0.10.0 ([[http://issues.apache.org/jira/browse/HADOOP-574|HADOOP-574]]),
but this had a few bugs so you should use Hadoop 0.10.1 or later.
+  * The S3 block filesystem was introduced in Hadoop 0.10.0 ([[http://issues.apache.org/jira/browse/HADOOP-574|HADOOP-574]]).
   * The S3 native filesystem was introduced in Hadoop 0.18.0 ([[http://issues.apache.org/jira/browse/HADOOP-930|HADOOP-930]])
and rename support was added in Hadoop 0.19.0 ([[https://issues.apache.org/jira/browse/HADOOP-3361|HADOOP-3361]]).
   * The S3A filesystem was introduced in Hadoop 2.6.0. Some issues were found and fixed for
later Hadoop versions[[https://issues.apache.org/jira/browse/HADOOP-11571|HADOOP-11571]],
so Hadoop-2.6.0's support of s3a must be considered an incomplete replacement for the s3n
FS.
  

Mime
View raw message