hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <ar...@yahoo-inc.com>
Subject Re: performance test tips?
Date Fri, 16 Nov 2007 18:44:58 GMT

On Fri, Nov 16, 2007 at 12:00:21PM -0600, jonathan doklovic wrote:
>We've finally got our hadoop cluster up, some data to crunch and a
>map/reduce job.
>After running a few configurations, i'm not sure about our performance
>and would like to get some advice....
>We have a 20 node ec2 cluster.
>We have 750MB of data.
>currently our job seems to be doing 1%/min on the cluster.
>Using a much smaller subset of data and running locally, the job takes a
>matter of seconds.
>Here's our hadoop-site.xml
>  <name>mapred.tasktracker.tasks.maximum</name>
>  <value>20</value>

That is very high, you are basically letting the TT spawn 40 child-jvms. I wouldn't go above
3 or 4 for that config.

Hopefully http://lucene.apache.org/hadoop/cluster_setup.html is useful...


View raw message