hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gian Lorenzo Thione <thi...@powerset.com>
Subject Re: Amazon EC2
Date Fri, 25 Aug 2006 23:04:16 GMT

At Powerset we have used EC2 and Hadoop with a large number of nodes,
successfully running Map/Reduce computations and HDFS. Pretty much like you
describe, we use HDFS for intermediate results and caching, and periodically
extract data to our local network. We are not really using S3 at the moment
for persistent storage.

A nice feature of Hadoop as measured against our use of EC2 has been the
capability of fluidly changing the number of instances that are part of the
cluster. Our instances are set up to join the cluster and the DFS as soon as
they are activated and when - for any reason - we lose those machines, the
overall process doesn't suffer. We have been quite happy with this, even at
significant number of instances.

As a byproduct of running these experiments, we have implemented some
patches to Hadoop to report to the master IP's and hostnames that are
different than the default ones assigned by InetAddress's static method
(getLocalHost). This is due to the fact that machines sometimes are assigned
to specific networks to deal with firewalls in various ways so we may want
to report Ips to the JobTracker from different interfaces, in order for the
tracker to contact them back. In our model the interface is specified as
part of the configuration parameters.

Is this something that the Hadoop project would be interested to

Lorenzo Thione

On 8/25/06 3:39 PM, "Doug Cutting" <cutting@apache.org> wrote:

> Has anyone tried running Hadoop on the Amazon Elastic Compute Cloud yet?
> http://www.amazon.com/gp/browse.html?node=201590011
> One way to use Hadoop on this would be to:
> 1. Allocate a pool of machines.
> 2. Start Hadoop daemons.
> 3. Load the HDFS filesystem with input from Amazon S3.
> 4. Run a series of MapReduce computations.
> 5. Copy the final output from HDFS back to Amazon S3.
> 6. Deallocate the machines.
> Steps (3) and (4) could be eliminated if a Hadoop FileSystem were
> implemented on S3, so that input and output could be accessed directly
> from S3.  (One might still use HDFS for intermediate data, as it should
> be faster.)
> The prices seem very reasonable.  100 nodes for 10 hours costs $100.
> Storing a terabyte on S3 costs $150/month.  Transferring a terabyte of
> offsite data (e.g., fetching 100M pages) costs $200.  So someone could
> use Nutch to keep a 100M page crawl, refreshed monthly, for around
> $500/month.  Such a crawl could be shared with other organizations who
> would themselves pay, a la carte, for their computations over it.
> If anyone tries Hadoop on EC2, please tell us how it goes.
> Doug

View raw message