Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>
Mime-Version: 1.0 (Apple Message framework v1278)
Content-Type: multipart/signed;
 boundary="Apple-Mail=_0D7DD151-E758-41A6-80AF-91DF9C181391";
 protocol="application/pgp-signature"; micalg=pgp-sha1
Subject: Re: Questions with regard to scheduling of map and reduce tasks
Date: Fri, 31 Aug 2012 15:59:39 -0700
In-Reply-To: 
 <CALg=eG91fP_nt5mJ5J=cT_xThyMZyONn2bmmJqv0XaYmO03qyw@mail.gmail.com>
To: user@hadoop.apache.org
References: 
 <CALg=eG_WbuX36ZNMPPnOOznzwm6sTYF0RhYFGSb6zmPaq8r2dQ@mail.gmail.com>
 <24F38861-6C09-4EDB-82FE-F2CD2B0390C4@hortonworks.com>
 <CALg=eG_oLf=kVuHg5swR2R0NnYK533mvWx85G8YYYDz8V_PRmw@mail.gmail.com>
 <93826E69-CBEE-4413-86C2-18368A3AF693@apache.org>
 <CALg=eG91fP_nt5mJ5J=cT_xThyMZyONn2bmmJqv0XaYmO03qyw@mail.gmail.com>
Message-Id: <B03CF76F-4B08-4611-8583-6B62ECAB50A7@apache.org>


--Apple-Mail=_0D7DD151-E758-41A6-80AF-91DF9C181391
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_73EDB25A-34DA-434D-AB7C-C38B44F81DAE"


--Apple-Mail=_73EDB25A-34DA-434D-AB7C-C38B44F81DAE
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=iso-8859-1


You don't need to touch the code related protocol-buffer records at all =
as there are java-native interfaces for everything, for e.g. - =
org.apache.hadoop.yarn.api.AMRMProtocol.

Regarding your question - The JobClient first obtains the locations of =
DFS blocks via the InputFormat.getSplits() and uploads the accumulated =
information into a split file, see Job.submitInternal() ->  =
JobSubmitter.writeSplits() ->  ...

The MR AM then downloads and reads the split file and reconstructs the =
splits-information and creates TaskAttempts(TAs) which then use it =
request containers. See MRAppMaster code: JobImpl.InitTransition for how =
TAs are created with host information.

HTH,

+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/

On Aug 31, 2012, at 4:17 AM, Vasco Visser wrote:

> Thanks again for the reply, it is becoming clear.
>=20
> While on the subject of going over the code, do you know by any chance
> where the piece of code is that creates resource requests according to
> locations of HDFS blocks? I am looking for that, but the protocol
> buffer stuff makes it difficult for me to understand what is going on.
>=20
> regards, Vasco
>=20

--Apple-Mail=_73EDB25A-34DA-434D-AB7C-C38B44F81DAE
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=iso-8859-1

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div><br></div><div>You don't need to touch the code related =
protocol-buffer records at all as there are java-native interfaces for =
everything, for e.g. -&nbsp;<span class=3D"Apple-style-span" =
style=3D"font-family: Monaco; font-size: 11px; =
">org.apache.hadoop.yarn.api.AMRMProtocol.</span></div><div><br></div><div=
>Regarding your question - The JobClient first obtains the locations of =
DFS blocks via the InputFormat.getSplits() and uploads the accumulated =
information into a split file, see Job.submitInternal() -&gt; =
&nbsp;JobSubmitter.writeSplits() -&gt; =
&nbsp;...</div><div><br></div><div>The MR AM then downloads and reads =
the split file and reconstructs the splits-information and creates =
TaskAttempts(TAs) which then use it request containers. See MRAppMaster =
code: JobImpl.InitTransition for how TAs are created with host =
information.</div><div><br></div><div =
apple-content-edited=3D"true"><div><div class=3D"">HTH,</div><div =
class=3D""><br></div><div class=3D"">+Vinod Kumar Vavilapalli</div><div =
class=3D"">Hortonworks Inc.<br class=3D""><a =
href=3D"http://hortonworks.com/" =
class=3D"">http://hortonworks.com/</a></div></div>
</div>

<br><div><div>On Aug 31, 2012, at 4:17 AM, Vasco Visser wrote:</div><br =
class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><div>Thanks =
again for the reply, it is becoming clear.<br><br>While on the subject =
of going over the code, do you know by any chance<br>where the piece of =
code is that creates resource requests according to<br>locations of HDFS =
blocks? I am looking for that, but the protocol<br>buffer stuff makes it =
difficult for me to understand what is going on.<br><br>regards, =
Vasco<br><br></div></blockquote></div></body></html>=

--Apple-Mail=_73EDB25A-34DA-434D-AB7C-C38B44F81DAE--

--Apple-Mail=_0D7DD151-E758-41A6-80AF-91DF9C181391
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
	filename=signature.asc
Content-Type: application/pgp-signature;
	name=signature.asc
Content-Description: Message signed with OpenPGP using GPGMail

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)

iQEcBAEBAgAGBQJQQUHcAAoJECVa31bDbF8PCEkIALkvbgDOh5xSm//ShPFdfaqt
dpF9ExZqihv1J6U6ggeEOgPMkjYooRVzSnznEJSPyMGRu2oggImNaxtOiKN9d5V9
Gn5tq+3dXca4erHfUM1M1Ff+MHD3hog4SaxE13bMbWRMusx+eFMY1Xne6nJih7zE
OThChy2QXyinIDeCIO/OwOiFHXydFJy59dg0ROEnL9f3K2N0Gjtxr5YMwIom96PV
cW2qK94mnj98tfR+QkXT1MhONLB5RnRLdifLIF0OX25R5mT1lY3T0sWnJNMpg8q7
SFLz/KbwMBgaZL873yan/e/4KQ+DrmmAZU5Cqbz14+YbNeCiSVfIthr1xcSMzoU=
=Wo9n
-----END PGP SIGNATURE-----

--Apple-Mail=_0D7DD151-E758-41A6-80AF-91DF9C181391--