Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of martin.dobmeier@gmail.com
 designates 74.125.83.48 as permitted sender)
MIME-Version: 1.0
Date: Thu, 13 Sep 2012 16:04:36 +0200
Message-ID: 
 <CA+gjK9AwMiTPBAUtX_w8SiQn0b7BBbra9rkdK8z_Ggswxq8aTQ@mail.gmail.com>
Subject: How does map-merge work exactly?
From: Martin Dobmeier <martin.dobmeier@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7b603c12a04dcc04c995c9bd

--047d7b603c12a04dcc04c995c9bd
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

I'm greatly confused about the spill/sort/merge thing going on during the
Map phase.

Here are some stats:
- io.sort.mb = 256 MB (80% spill threshold)
- io.sort.factor = 64
- spills performed during Map: 117
- number of reducers: 96

Now I'm having real trouble understanding the following log output.

...
mapred.Merger: Merging 117 sorted segments
mapred.Merger: Down to the last merge-pass, with 0 segments left of total
size: 0 bytes
...
mapred.Merger: Merging 117 sorted segments
mapred.Merger: Merging 54 intermediate segments out of a total of 56
mapred.Merger: Down to the last merge-pass, with 3 segments left of total
size: 67119046 bytes
...
mapred.Merger: Merging 117 sorted segments
mapred.Merger: Merging 54 intermediate segments out of a total of 117
mapred.Merger: Down to the last merge-pass, with 64 segments left of total
size: 1609011189 bytes
...

What exactly is a segment? Is it the number of spills?
What does "0 segments left" mean? Does it mean that the merge could be
performed on the first pass?
Why are only 54 segments merged instead of "io.sort.factor" segments?
(io.sort.factor determines the number of files to merge during a pass,
right?)
Why is the merge performed "number of reducers" times? (I'm counting the
phrase "Merging 117 segments" exactly 96 times)

Thanks a lot!
Martin

--047d7b603c12a04dcc04c995c9bd
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi all,<br><br>I&#39;m greatly confused about the spill/sort/merge thing go=
ing on during the Map phase.<br><br>Here are some stats:<br>- io.sort.mb =
=3D 256 MB (80% spill threshold)<br>- io.sort.factor =3D 64<br>- spills per=
formed during Map: 117<br>


- number of reducers: 96<br><br>Now I&#39;m having real trouble understandi=
ng the following log output.<br><br>...<br>mapred.Merger: Merging 117 sorte=
d segments<br>mapred.Merger: Down to the last merge-pass, with 0 segments l=
eft of total size: 0 bytes<br>


...<br>mapred.Merger: Merging 117 sorted segments<br>mapred.Merger: Merging=
 54 intermediate segments out of a total of 56<br>
mapred.Merger: Down to the last merge-pass, with 3 segments left of total s=
ize: 67119046 bytes<br>...<br>mapred.Merger: Merging 117 sorted segments<br=
>mapred.Merger: Merging 54 intermediate segments out of a total of 117<br>


mapred.Merger: Down to the last merge-pass, with 64 segments left of total =
size: 1609011189 bytes<br>...<br><br>What exactly is a segment? Is it the n=
umber of spills?<br>What does &quot;0 segments left&quot; mean? Does it mea=
n that the merge could be performed on the first pass?<br>


Why are only 54 segments merged instead of &quot;io.sort.factor&quot; segme=
nts? (io.sort.factor determines the number of files to merge during a pass,=
 right?)<br>Why is the merge performed &quot;number of reducers&quot; times=
? (I&#39;m counting the phrase &quot;Merging 117 segments&quot; exactly 96 =
times)<br>


<br>Thanks a lot!<br>Martin<br><br>

--047d7b603c12a04dcc04c995c9bd--