hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Scott Carey <sc...@richrelevance.com>
Subject Re: hadoop idle time on terasort
Date Mon, 07 Dec 2009 21:57:30 GMT

On 12/2/09 12:22 PM, "Vasilis Liaskovitis" <vliaskov@gmail.com> wrote:

> Hi,
> I am using hadoop-0.20.1 to run terasort and randsort benchmarking
> tests on a small 8-node linux cluster. Most runs consist of usually
> low (<50%) core utilizations in the map and reduce phase, as well as
> heavy I/O phases . There is usually a large fraction of runtime for
> which cores are idling and i/o disk traffic is not heavy.
> On average for the duration of a terasort run I get 20-30% cpu
> utilization, 10-30% iowait times and the rest 40-70% is idle time.
> This is data collected with mpstat for the duration of the run across
> the cores of a specific node. This utilization behaviour is true and
> symmetric for all tasktracker/data nodes (The namenode cores and I/O
> are mostly idle, so there doesn┬╣t seem to be a bottleneck in the
> namenode).

Look at individual task logs for map and reduce through the UI.  Also, look
at the cluster utilization during a run -- are most map and reduce slots
full most of the time, or is the slot utilization low?

Running the fair scheduler -- or any scheduler -- that can be configured to
schedule more than one task per heartbeat can dramatically increase your
slot utilization if it is low.

Next, if you find that your delay times correspond with the shuffle phase
(look in the reduce logs), there are fixes in 0.21 for that on the way, but
there is a quick win, one line change that cuts shuffle times down a lot on
clusters that have a large ratio of map tasks per node if the map output is
not too large.  For a pure sort test, the map outputs are medium sized (the
same size as the input), so this might not help.  But the indicators of the
issue are in the reduce task logs.
See this ticket: http://issues.apache.org/jira/browse/MAPREDUCE-318 and my
comment from June 10 2009.

My summary is that the hadoop scheduling process has not been so far for
servers that can run more than 6 or so tasks at once.  A server capable of
running 12 maps is especially prone to running under-utilized.  Many changes
in the 0.21 timeframe address some of this.

> thanks,
> - Vasilis

View raw message