Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of amitmittal5@gmail.com
 designates 209.85.216.50 as permitted sender)
MIME-Version: 1.0
Date: Mon, 27 Jan 2014 17:47:26 +0530
Message-ID: 
 <CAB8BrSs6Dsyrjhie9XBv9iNv7sfXANQiR+TdwLDag_d0=0h79w@mail.gmail.com>
Subject: Does all reducer take input from all NodeManager/Tasktrackers of Map
 tasks
From: Amit Mittal <amitmittal5@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a11345524d36b6404f0f2b00f

--001a11345524d36b6404f0f2b00f
Content-Type: text/plain; charset=ISO-8859-1

Hi,

Does all reducer take input from all NodeManager/Tasktrackers of Map tasks ?

*Reference:* "Hadoop: The Definitive Guide:3rd Ed" book by "Tom White"
On page# 210 (Ch 6: How MapReduce Works > Shuffle & Sort > The reducer side)

There is a note, here is the text from book:
How do reducers know which machines to fetch map output from?
...
Therefore, for a given job, the jobtracker (or application master) knows
the mapping between map outputs and hosts. A thread in the reducer
periodically asks the master for map output hosts
until it has retrieved them all.
...
*Question 1:* I believe the TaskTracker and then JobTracker/AppMaster will
receive the updates through call to Task.statusUpdate(TaskUmbilicalProtocol
obj). By which the JobTracker/AM will know the location of the map's o/p
file and host details etc, however how it will know what all the partitions
or keys this output has. In other words, from the heartbeat, how JobTracker
will know about data partitions/keys? It will be required to decide from
which Mapper, the mapper's output needs to be pulled or not.
*Question 2:* In short, not all reducer takes output from all Mappers, they
only connects and takes output related to the keys partitioned for that
particular reducer.

Thanks
Amit Mittal

--001a11345524d36b6404f0f2b00f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div style=3D"font-family:arial,sans-serif;font-size:13px"=
>Hi,</div><div style=3D"font-family:arial,sans-serif;font-size:13px"><br></=
div><div style=3D"font-family:arial,sans-serif;font-size:13px">Does all red=
ucer take input from all NodeManager/Tasktrackers of Map tasks ?</div>
<div style=3D"font-family:arial,sans-serif;font-size:13px"><br></div><div s=
tyle=3D"font-family:arial,sans-serif;font-size:13px"><b>Reference:</b>=A0&q=
uot;Hadoop: The Definitive Guide:3rd Ed&quot; book by &quot;Tom White&quot;=
</div>
<div style=3D"font-family:arial,sans-serif;font-size:13px"><div>On page# 21=
0 (Ch 6: How MapReduce Works &gt; Shuffle &amp; Sort &gt; The reducer side)=
</div><div><br></div><div>There is a note, here is the text from book:=A0</=
div>
<div>How do reducers know which machines to fetch map output from?</div><di=
v>...</div><div><div>Therefore, for a given job, the jobtracker (or applica=
tion master) knows the mapping between map outputs and hosts. A thread in t=
he reducer periodically asks the master for map output hosts</div>
<div>until it has retrieved them all.</div></div><div>...</div><div><b>Ques=
tion 1:</b>=A0I believe the TaskTracker and then JobTracker/AppMaster will =
receive the updates through call to Task.statusUpdate(TaskUmbilicalProtocol=
 obj). By which the JobTracker/AM will know the location of the map&#39;s o=
/p file and host details etc, however how it will know what all the partiti=
ons or keys this output has. In other words, from the heartbeat, how JobTra=
cker will know about data partitions/keys? It will be required to decide fr=
om which Mapper, the mapper&#39;s output needs to be pulled or not.</div>
<div><b>Question 2:</b>=A0In short, not all reducer takes output from all M=
appers, they only connects and takes output related to the keys partitioned=
 for that particular reducer.</div><div><br></div><div>Thanks</div><div>Ami=
t Mittal</div>
</div></div>

--001a11345524d36b6404f0f2b00f--