Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of vinodkv@hortonworks.com
 designates 209.85.220.51 as permitted sender)
From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>
Mime-Version: 1.0 (Apple Message framework v1283)
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_3FB34611-57C2-4350-8B16-2296B3766253"
Subject: Re: MapReduce - FileInputFormat and Locality
Date: Wed, 8 May 2013 15:00:13 -0700
In-Reply-To: <518A5F6D.30700@etinternational.com>
To: user@hadoop.apache.org
References: <518A5F6D.30700@etinternational.com>
Message-Id: <9AA3D4E4-CDCB-4BDC-94F2-77D8CB3FD0CE@apache.org>


--Apple-Mail=_3FB34611-57C2-4350-8B16-2296B3766253
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=iso-8859-1

I think you misread it.

If a given split has only one block, it uses all the locations of that =
block.

If it so happens that a given split has multiple blocks, it uses all the =
locations of the first block.

HTH,
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/


On May 8, 2013, at 7:21 AM, Brian C. Huffman wrote:

> All,
>=20
> I'm trying to understand how the current FileInputFormat implements =
locality.  As far as I can tell, it calculates splits using getSplit and =
each split will contain the node that hosts the first block of data in =
that split.  Is my understanding correct?
>=20
> Looking at the FileInputFormat for the old API (mapred), it appears =
that it does more to implement locality, using getSplitHosts to "return =
the hosts that contribute most for a given split"
>=20
> If I understand correctly, why was this changed?
>=20
> Thanks,
> Brian
>=20


--Apple-Mail=_3FB34611-57C2-4350-8B16-2296B3766253
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=iso-8859-1

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div>I think you misread it.</div><div><br></div><div>If a given split =
has only one block, it uses all the locations of that =
block.</div><div><br></div><div>If it so happens that a given split has =
multiple blocks, it uses all the locations of the first =
block.</div><br><div>
<span class=3D"Apple-style-span" style=3D"border-collapse: separate; =
color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; =
font-variant: normal; font-weight: normal; letter-spacing: normal; =
line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: =
0px; text-transform: none; white-space: normal; widows: 2; word-spacing: =
0px; -webkit-border-horizontal-spacing: 0px; =
-webkit-border-vertical-spacing: 0px; =
-webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span =
class=3D"Apple-style-span" style=3D"border-collapse: separate; color: =
rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: =
normal; font-weight: normal; letter-spacing: normal; line-height: =
normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; =
text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: =
0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div =
style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space; "><span class=3D"Apple-style-span" =
style=3D"border-collapse: separate; color: rgb(0, 0, 0); font-family: =
Helvetica; font-style: normal; font-variant: normal; font-weight: =
normal; letter-spacing: normal; line-height: normal; orphans: 2; =
text-align: -webkit-auto; text-indent: 0px; text-transform: none; =
white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: =
0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div =
style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; =
-webkit-line-break: after-white-space; "><div =
class=3D"">HTH,</div></div></span></div></span></span></div><div =
apple-content-edited=3D"true"><span class=3D"Apple-style-span" =
style=3D"border-collapse: separate; color: rgb(0, 0, 0); font-family: =
Helvetica; font-style: normal; font-variant: normal; font-weight: =
normal; letter-spacing: normal; line-height: normal; orphans: 2; =
text-align: -webkit-auto; text-indent: 0px; text-transform: none; =
white-space: normal; widows: 2; word-spacing: 0px; =
-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: =
0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: =
auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div><div =
class=3D"">+Vinod Kumar Vavilapalli</div><div class=3D"">Hortonworks =
Inc.<br class=3D""><a href=3D"http://hortonworks.com/" =
class=3D"">http://hortonworks.com/</a></div></div><div =
class=3D""><br></div></span>
</div>
<br><div><div>On May 8, 2013, at 7:21 AM, Brian C. Huffman =
wrote:</div><br class=3D"Apple-interchange-newline"><blockquote =
type=3D"cite"><div>All,<br><br>I'm trying to understand how the current =
FileInputFormat implements locality. &nbsp;As far as I can tell, it =
calculates splits using getSplit and each split will contain the node =
that hosts the first block of data in that split. &nbsp;Is my =
understanding correct?<br><br>Looking at the FileInputFormat for the old =
API (mapred), it appears that it does more to implement locality, using =
getSplitHosts to "return the hosts that contribute most for a given =
split"<br><br>If I understand correctly, why was this =
changed?<br><br>Thanks,<br>Brian<br><br></div></blockquote></div><br></bod=
y></html>=

--Apple-Mail=_3FB34611-57C2-4350-8B16-2296B3766253--