Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: "Ravi  Gummadi" <gravi@yahoo-inc.com>
To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
Date: Thu, 23 Dec 2010 02:37:52 +0530
Subject: Re: Spill and Map Output
Thread-Topic: Spill and Map Output
Thread-Index: AcuiGZZ/gRWTlXKaRIqgZ4r00AC9qgAA/xC/
Message-ID: <C93868A5.238F5%gravi@yahoo-inc.com>
In-Reply-To: <AANLkTikQOvBp1EDrfBccFnoziQB9WAXFiwg8NPMn0B1q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C93868A5238F5graviyahooinccom_"
MIME-Version: 1.0

--_000_C93868A5238F5graviyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Each map task will generate a single intermediate file (i.e. Map output fil=
e). This is obtained by merging multiple spills, if spills needed to happen=
.

Index file gives the details of the offset and length for each reducer. Off=
set is offset in the map output file where the input data for the particula=
r reducer starts and length is the size of the data starting from the offse=
t.

-Ravi


On 12/23/10 2:17 AM, "Pedro Costa" <psdc1978@gmail.com> wrote:

Hi,

1 - I would like to understand how a partition works in the Map
Reduce. I know that the Map Reduce contains the IndexRecord class that
indicates the length of something. Is it the length of a partition or
of a spill?

2 - In large map output, a partition can be a set of spills, or a
spill is simple the same thing as a partition?

Thanks,
--
Pedro


--_000_C93868A5238F5graviyahooinccom_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML>
<HEAD>
<TITLE>Re: Spill and Map Output</TITLE>
</HEAD>
<BODY>
<FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:=
11pt'>Each map task will generate a single intermediate file (i.e. Map outp=
ut file). This is obtained by merging multiple spills, if spills needed to =
happen.<BR>
<BR>
Index file gives the details of the offset and length for each reducer. Off=
set is offset in the map output file where the input data for the particula=
r reducer starts and length is the size of the data starting from the offse=
t.<BR>
<BR>
-Ravi<BR>
<BR>
<BR>
On 12/23/10 2:17 AM, &quot;Pedro Costa&quot; &lt;<a href=3D"psdc1978@gmail.=
com">psdc1978@gmail.com</a>&gt; wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"=
><SPAN STYLE=3D'font-size:11pt'>Hi,<BR>
<BR>
1 - I would like to understand how a partition works in the Map<BR>
Reduce. I know that the Map Reduce contains the IndexRecord class that<BR>
indicates the length of something. Is it the length of a partition or<BR>
of a spill?<BR>
<BR>
2 - In large map output, a partition can be a set of spills, or a<BR>
spill is simple the same thing as a partition?<BR>
<BR>
Thanks,<BR>
--<BR>
Pedro<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE>
</BODY>
</HTML>


--_000_C93868A5238F5graviyahooinccom_--