Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=message-id:from:to:in-reply-to:content-type:mime-version:
	subject:date:references:x-mailer;
	b=H+mFwU/eyWhglxdO4zKPm2PdK0fLJ8b24BcFosC0D3ahjSGStQRWvrC+rfjb4Ai1
Message-Id: <C54456F1-0EE5-4265-BA86-48FE04A27C87@yahoo-inc.com>
From: Arun C Murthy <acm@yahoo-inc.com>
To: core-user@hadoop.apache.org
In-Reply-To: <c9b0d8bd0904192126p54766564s507e8861455f2e7b@mail.gmail.com>
Content-Type: multipart/alternative; boundary=Apple-Mail-1--59118047
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: Performance question
Date: Mon, 20 Apr 2009 20:54:46 +0530
References: <c9b0d8bd0904192126p54766564s507e8861455f2e7b@mail.gmail.com>

--Apple-Mail-1--59118047
Content-Type: text/plain;
	charset=WINDOWS-1252;
	format=flowed;
	delsp=yes
Content-Transfer-Encoding: quoted-printable


On Apr 20, 2009, at 9:56 AM, Mark Kerzner wrote:

> Hi,
>
> I ran a Hadoop MapReduce task in the local mode, reading and writing =20=

> from
> HDFS, and it took 2.5 minutes. Essentially the same operations on =20
> the local
> file system without MapReduce took 1/2 minute.  Is this to be =20
> expected?
>

Hmm... some overhead is expected, but this seems too much. What =20
version of Hadoop are your running?

It's hard to help without more details about your application, =20
configuration etc., I'll try...


> It seemed that the system lost most of the time in the MapReduce =20
> operation,
> such as after these messages
>
> 09/04/19 23:23:01 INFO mapred.LocalJobRunner: reduce > reduce
> 09/04/19 23:23:01 INFO mapred.JobClient:  map 100% reduce 92%
> 09/04/19 23:23:04 INFO mapred.LocalJobRunner: reduce > reduce
>
> it waited for a long time. The final output lines were
>

It could either be the reduce-side merge or the hdfs-write. Can you =20
check your task-logs and data-node logs?

> 09/04/19 23:24:13 INFO mapred.JobClient:     Combine input records=3D185=

> 09/04/19 23:24:13 INFO mapred.JobClient:     Combine output =20
> records=3D185

That shows that the combiner is useless for this app, turn it off - it =20=

adds unnecessary overhead.

> 09/04/19 23:24:13 INFO mapred.JobClient:   File Systems
> 09/04/19 23:24:13 INFO mapred.JobClient:     HDFS bytes read=3D138103444=

> 09/04/19 23:24:13 INFO mapred.JobClient:     HDFS bytes =20
> written=3D107357785
> 09/04/19 23:24:13 INFO mapred.JobClient:     Local bytes =20
> read=3D282509133
> 09/04/19 23:24:13 INFO mapred.JobClient:     Local bytes =20
> written=3D376697552

For the amount of data you are processing, you are doing far too much =20=

local-disk i/o.
'Local bytes written' should be _very_ close to the 'Map output bytes' =20=

i.e 91M for 'maps' and zero for reduces. (See the counters-table on =20
the job-details web-ui.)

There are a few knobs you need to tweak to get closer to more optimal =20=

performance, the good news is that it's doable - the bad news is that =20=

one _has_ to get his/her fingers dirty...

Some knobs you will be interested in are:

Map-side:
=95io.sort.mb
=95io.sort.factor
=95io.sort.record.percent
=95io.sort.spill.percent

  Reduce-side
* mapred.reduce.parallel.copies
* mapred.reduce.copy.backoff
* mapred.job.shuffle.input.buffer.percent
* mapred.job.shuffle.merge.percent
* mapred.inmem.merge.threshold
* mapred.job.reduce.input.buffer.percent


Check description for each of them in hadoop-default.xml or mapred-=20
default.xml (depending on the version of Hadoop you are running).
Some more details available here: =
http://wiki.apache.org/hadoop-data/attachments/HadoopPresentations/attachm=
ents/TuningAndDebuggingMapReduce_ApacheConEU09.pdf

hth,
Arun


--Apple-Mail-1--59118047--