hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nate Carlson <natec...@natecarlson.com>
Subject Re: Hadoop performance on EC2?
Date Fri, 11 Apr 2008 21:01:56 GMT
On Wed, 9 Apr 2008, Chris K Wensel wrote:
> make sure all nodes are running in the same 'availability zone', 
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1347


> and that you are using the new xen kernels.
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1353&categoryID=101
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1354&categoryID=101


> also, make sure each node is addressing its peers via the ec2 private 
> addresses, not the public ones.


> there is a patch in jira for the ec2/contrib scripts that address these 
> issues.
> https://issues.apache.org/jira/browse/HADOOP-2410
> if you use those scripts, you will be able to see a ganglia display 
> showing utilization on the machines. 8/7 map/reducers sounds like alot.

Reduced - I dropped it to 3/2 for testing.

I am using these scripts now, and am still seeing very poor performance on 
EC2 compared to my development environment.  ;(

I'll be capturing some more extensive stats over the weekend, and see if I 
can glean anything useful...

| nate carlson | natecars@natecarlson.com | http://www.natecarlson.com |
|       depriving some poor village of its idiot since 1981            |

View raw message