hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Lucene-hadoop Wiki] Update of "AmazonEC2" by TomWhite
Date Thu, 18 Jan 2007 09:48:33 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.

The following page has been changed by TomWhite:
http://wiki.apache.org/lucene-hadoop/AmazonEC2

------------------------------------------------------------------------------
  
  == Future Work ==
  
- Ideally Hadoop could directly access job data from [http://www.amazon.com/gp/browse.html?node=16427261
Amazon S3] (Simple Storage Service).  Initial input could be read from S3 when a cluster is
launched, and the final output could be written back to S3 before the cluster is decomissioned.
 Intermediate, temporary data, only needed between MapReduce passes, would be more efficiently
stored in Hadoop's DFS.  This would require an implementation of a Hadoop [http://lucene.apache.org/hadoop/docs/api/org/apache/hadoop/fs/FileSystem.html
FileSystem] for S3.  There are two issues in Hadoop's bug database related to this:
+ Ideally Hadoop could directly access job data from [http://www.amazon.com/gp/browse.html?node=16427261
Amazon S3] (Simple Storage Service).  Initial input could be read from S3 when a cluster is
launched, and the final output could be written back to S3 before the cluster is decomissioned.
 Intermediate, temporary data, only needed between MapReduce passes, would be more efficiently
stored in Hadoop's DFS. From Hadoop 0.10.1 onwards there is an implementation of a Hadoop
[http://lucene.apache.org/hadoop/docs/api/org/apache/hadoop/fs/FileSystem.html FileSystem]
for S3. See ["AmazonS3"].
  
-  * [http://issues.apache.org/jira/browse/HADOOP-574 HADOOP-574]
-  * [http://issues.apache.org/jira/browse/HADOOP-571 HADOOP-571]
- 
- Please vote for these issues in Jira if you feel this would help your project.  (Anyone
can create themselves a Jira account in order to vote on issues, etc.)
- 
- [[Anchor(AutomatedScripts])]]
+ [[Anchor(AutomatedScripts)]]
  = Automated Scripts =
  
  == Setting up ==

Mime
View raw message