Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of vliaskov@gmail.com designates
 209.85.160.50 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type
         :content-transfer-encoding;
        b=D2G3veF4taqr1PnW9W4jlVv5Gj7nu9EZ+NNXkRCRDUmo+ltwibTEw1vCqavafA+JSK
         ovnNHGrPdVisHjGo5WS0fuaPpS9QXED5029XtAn061sX7AxsDGgP045uOxGfzCVikMY4
         tOTn5oy3aic8R/iny6YxBb1bS+q34NLZLAjps=
MIME-Version: 1.0
Date: Wed, 2 Dec 2009 14:22:12 -0600
Message-ID: <bec373e0912021222g6a38de78r91d92501faac91e2@mail.gmail.com>
Subject: hadoop idle time on terasort
From: Vasilis Liaskovitis <vliaskov@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Hi,

I am using hadoop-0.20.1 to run terasort and randsort benchmarking
tests on a small 8-node linux cluster. Most runs consist of usually
low (<50%) core utilizations in the map and reduce phase, as well as
heavy I/O phases . There is usually a large fraction of runtime for
which cores are idling and i/o disk traffic is not heavy.

On average for the duration of a terasort run I get 20-30% cpu
utilization, 10-30% iowait times and the rest 40-70% is idle time.
This is data collected with mpstat for the duration of the run across
the cores of a specific node. This utilization behaviour is true and
symmetric for all tasktracker/data nodes (The namenode cores and I/O
are mostly idle, so there doesn=92t seem to be a bottleneck in the
namenode).

I am looking for an explanation for the significant idle-time in the
runs. Could it have something to do with misconfigured network/RPC
latency hadoop paremeters? For example, I have tried to increase
mapred.heartbeats.in.second to 1000 from 100 but that didn=92t help. The
network bandwidth (1Gige card on each node) is not saturated during
the runs, according to my netstat results.

Have other people noticed significant cpu idle times that can=92t be
explained by I/O traffic?

Is it reasonable to always expect decreasing idle times as the
terasort dataset scales on the same cluster? I =91ve only tried 2 small
datasets of 40GB and 64GB each, but core utilizations didn=92t increase
with the runs done so far.

Yahoo=92s paper on terasort (http://sortbenchmark.org/Yahoo2009.pdf)
mentions several performance optimizations, some of which seem
relevant to idle times. I am wondering which, if any, of the yahoo
patches are part of the hadoop-0.20.1 distribution.

Would it be a good idea to try a development version of hadoop to
resolve this issue?

thanks,

- Vasilis