Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of liyin.liangly@aliyun-inc.com
 designates 110.75.170.56 as permitted sender)
From: =?gb2312?B?wbrA7tOh?= <liyin.liangly@aliyun-inc.com>
To: <common-user@hadoop.apache.org>
References: 
 <CA+Omw9hz_z0YUtqLAGmktiqVGAZJhnGN5qSoq2btEr9vE9T5yQ@mail.gmail.com>
In-Reply-To: 
 <CA+Omw9hz_z0YUtqLAGmktiqVGAZJhnGN5qSoq2btEr9vE9T5yQ@mail.gmail.com>
Subject: =?gb2312?B?tPC4tDogTWFwUmVkdWNlIHNodWZmbGUgcXVlc3Rpb24=?=
Date: Fri, 3 Aug 2012 22:33:48 +0800
Message-ID: <001301cd7184$fe88eaa0$fb9abfe0$@liangly@aliyun-inc.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="gb2312"
Content-Transfer-Encoding: quoted-printable
Thread-Index: Ac1xez4ToRcupKl/RPGCJ/ITx5/BUgACQ6SQ
Content-Language: zh-cn

When a map task is done, its output is always flushed to the disk and =
merged
to one file.
The benefit is that if the reducer is failed, the map need not to =
re-run.

Liyin Liang

-----=D3=CA=BC=FE=D4=AD=BC=FE-----
=B7=A2=BC=FE=C8=CB: Satheesh Kumar [mailto:nkseam@gmail.com]=20
=B7=A2=CB=CD=CA=B1=BC=E4: 2012=C4=EA8=D4=C23=C8=D5 21:23
=CA=D5=BC=FE=C8=CB: common-user@hadoop.apache.org
=D6=F7=CC=E2: MapReduce shuffle question

Team, can someone please clarify the following question?

In the map phase, the map output is written to the local disk. And in =
the
shuffle phase, the map output partitions are transferred to reduce nodes
using http. So, my question is assuming there are no spills (data set is
small enough to accommodate this), will the map output be transferred
directly from memory to the reduce nodes using http without a disk =
access
to write the map output? Or, is the map output always flushed to the =
disk
before transferred to reduce nodes?

Appreciate the help.

Thanks,
Satheesh