hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vasilis Liaskovitis <vlias...@gmail.com>
Subject Re: hadoop idle time on terasort
Date Thu, 03 Dec 2009 00:37:57 GMT
Hi Todd,

thanks for the reply.

> This is seen reasonably often, and could be partly due to missed
> configuration changes. A few things to check:
> - Did you increase the number of tasks per node from the default? If you
> have a reasonable number of disks/cores, you're going to want to run a lot
> more than 2 map and 2 reduce tasks on each node.

For all tests so far, I have increased
mapred.tasktracker.reduce.tasks.maximum to the number of cores per
tasktracker/node. (12 cores per node).
I 've also set mapred.map.tasks and mapred.reduce.tasks to a prime
close to the number of nodes i.e. 8. (though the recommendation for
mapred.map.tasks is "a prime several times greater than the number of

> - Have you tuned any other settings? If you google around you can find some
> guides for configuration tuning that should help squeeze some performance
> out of your cluster.

I am reusing JVMs. I also enabled default codec compression (native
zlib I think) for intermediate map outputs. This decreased iowait
times for some datasets. But idle time is still significant even with
compression. I wonder if LZO compression would have better results -
less overall execution time and perhaps less idle time?

I also increased io.sort.mb (set to half the JVM heapsize) though I am
not sure how that affected performance yet. If other parameters could
be significant here, let me know. Would increasing the number of i/o
streams (io.sort.factor I think) help, with a not-so-beefy disk system
per node?

If you can recommend specific tutorial/guide/blog for performance
tuning, fell free to share.(though I suspect there may be so many out

> There are several patches that aren't in 0.20.1 but will be in 0.21 that
> help performance. These aren't eligible for backport into 0.20 since point
> releases are for bug fixes only. Some are eligible for backporting into
> Cloudera's distro (or Yahoo's) and may show up in our next release (CDH3)
> which should be available first in January for those who like to live on the
> edge.

ok thanks. I 'll try to check out 0.21 or a cloudera distro at some
point. I wonder if there's a cetralized svn/git somewhere if I want to
build from source. Or do I need to somehow combine all subprojects
hadoop-common, hadoop-mapred and hadoop-hdfs?

thanks again,

- Vasilis

> Thanks,
> -Todd
> On Wed, Dec 2, 2009 at 12:22 PM, Vasilis Liaskovitis <vliaskov@gmail.com>wrote:
>> Hi,
>> I am using hadoop-0.20.1 to run terasort and randsort benchmarking
>> tests on a small 8-node linux cluster. Most runs consist of usually
>> low (<50%) core utilizations in the map and reduce phase, as well as
>> heavy I/O phases . There is usually a large fraction of runtime for
>> which cores are idling and i/o disk traffic is not heavy.
>> On average for the duration of a terasort run I get 20-30% cpu
>> utilization, 10-30% iowait times and the rest 40-70% is idle time.
>> This is data collected with mpstat for the duration of the run across
>> the cores of a specific node. This utilization behaviour is true and
>> symmetric for all tasktracker/data nodes (The namenode cores and I/O
>> are mostly idle, so there doesn’t seem to be a bottleneck in the
>> namenode).
>> I am looking for an explanation for the significant idle-time in the
>> runs. Could it have something to do with misconfigured network/RPC
>> latency hadoop paremeters? For example, I have tried to increase
>> mapred.heartbeats.in.second to 1000 from 100 but that didn’t help. The
>> network bandwidth (1Gige card on each node) is not saturated during
>> the runs, according to my netstat results.
>> Have other people noticed significant cpu idle times that can’t be
>> explained by I/O traffic?
>> Is it reasonable to always expect decreasing idle times as the
>> terasort dataset scales on the same cluster? I ‘ve only tried 2 small
>> datasets of 40GB and 64GB each, but core utilizations didn’t increase
>> with the runs done so far.
>> Yahoo’s paper on terasort (http://sortbenchmark.org/Yahoo2009.pdf)
>> mentions several performance optimizations, some of which seem
>> relevant to idle times. I am wondering which, if any, of the yahoo
>> patches are part of the hadoop-0.20.1 distribution.
>> Would it be a good idea to try a development version of hadoop to
>> resolve this issue?
>> thanks,
>> - Vasilis

View raw message