Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of julianbui@gmail.com designates
 209.85.217.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <83C2D687-B3E3-45F3-A36A-9F1AFE1FE6AD@gmail.com>
References: 
 <CAFWc0y0v9Qev82_6Ge1Qv7J=ye_aAtByEccz419-sZpeoPVi6A@mail.gmail.com>
	<83C2D687-B3E3-45F3-A36A-9F1AFE1FE6AD@gmail.com>
Date: Tue, 5 Mar 2013 15:50:37 -0800
Message-ID: 
 <CAFWc0y3my=0xaWXCO9kvT9JiLpf0w8NMKyNuxkcsOqE2Cp8g4Q@mail.gmail.com>
Subject: Re: basic question about rack awareness and computation migration
From: Julian Bui <julianbui@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=f46d042c6423edfdac04d73623c6

--f46d042c6423edfdac04d73623c6
Content-Type: text/plain; charset=ISO-8859-1

Hi Rohit,

Thanks for responding.

> a task can be scheduled by hadoop to be executed on the same node which
is having data.

In my case, the mapper won't actually know where the data resides at the
time of being scheduled.  It only knows what data it will be accessing when
it reads in the keys.  In other words, the task will be already be running
by the time the mapper figures out what data must be accessed - so how can
hadoop know where to execute the code?

I'm still lost.  Please help if you can.

-Julian

On Tue, Mar 5, 2013 at 11:15 AM, Rohit Kochar <mnit.rohit@gmail.com> wrote:

> Hello ,
> To be precise this is hidden from the developer and you need not write any
> code for this.
> Whenever any file is stored in HDFS than it is splitted into block size of
> configured size and each block could potentially be stored on different
> datanode.All this information of which file contains which blocks resides
> with the namenode.
>
> So essentially whenever a file is accessed via DFS Client it requests the
>  NameNode for metadata,
> which DFS client uses to provide the file in streaming fashion to enduser.
>
> Since namenode knows the location of all the blocks/files ,a task can be
> scheduled by hadoop to be executed on the same node which is having data.
>
> Thanks
> Rohit Kochar
>
> On 05-Mar-2013, at 5:19 PM, Julian Bui wrote:
>
> > Hi hadoop users,
> >
> > I'm trying to find out if computation migration is something the
> developer needs to worry about or if it's supposed to be hidden.
> >
> > I would like to use hadoop to take in a list of image paths in the hdfs
> and then have each task compress these large, raw images into something
> much smaller - say jpeg  files.
> >
> > Input: list of paths
> > Output: compressed jpeg
> >
> > Since I don't really need a reduce task (I'm more using hadoop for its
> reliability and orchestration aspects), my mapper ought to just take the
> list of image paths and then work on them.  As I understand it, each image
> will likely be on multiple data nodes.
> >
> > My question is how will each mapper task "migrate the computation" to
> the data nodes?  I recall reading that the namenode is supposed to deal
> with this.  Is it hidden from the developer?  Or as the developer, do I
> need to discover where the data lies and then migrate the task to that
> node?  Since my input is just a list of paths, it seems like the namenode
> couldn't really do this for me.
> >
> > Another question: Where can I find out more about this?  I've looked up
> "rack awareness" and "computation migration" but haven't really found much
> code relating to either one - leading me to believe I'm not supposed to
> have to write code to deal with this.
> >
> > Anyway, could someone please help me out or set me straight on this?
> >
> > Thanks,
> > -Julian
>
>

--f46d042c6423edfdac04d73623c6
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Rohit,=A0<div><br></div><div>Thanks for responding.</div><div><br></div>=
<div>&gt;=A0a task can be scheduled by hadoop to be executed on the same no=
de which is having data.<div><br></div><div>In my case, the mapper won&#39;=
t actually know where the data resides at the time of being scheduled. =A0I=
t only knows what data it will be accessing when it reads in the keys. =A0I=
n other words, the task will be already be running by the time the mapper f=
igures out what data must be accessed - so how can hadoop know where to exe=
cute the code?</div>
<div><br></div><div>I&#39;m still lost. =A0Please help if you can.</div><di=
v><br></div><div>-Julian<br><br><div class=3D"gmail_quote">On Tue, Mar 5, 2=
013 at 11:15 AM, Rohit Kochar <span dir=3D"ltr">&lt;<a href=3D"mailto:mnit.=
rohit@gmail.com" target=3D"_blank">mnit.rohit@gmail.com</a>&gt;</span> wrot=
e:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hello ,<br>
To be precise this is hidden from the developer and you need not write any =
code for this.<br>
Whenever any file is stored in HDFS than it is splitted into block size of =
configured size and each block could potentially be stored on different dat=
anode.All this information of which file contains which blocks resides with=
 the namenode.<br>

<br>
So essentially whenever a file is accessed via DFS Client it requests the =
=A0NameNode for metadata,<br>
which DFS client uses to provide the file in streaming fashion to enduser.<=
br>
<br>
Since namenode knows the location of all the blocks/files ,a task can be sc=
heduled by hadoop to be executed on the same node which is having data.<br>
<br>
Thanks<br>
<span class=3D"HOEnZb"><font color=3D"#888888">Rohit Kochar<br>
</font></span><div class=3D"HOEnZb"><div class=3D"h5"><br>
On 05-Mar-2013, at 5:19 PM, Julian Bui wrote:<br>
<br>
&gt; Hi hadoop users,<br>
&gt;<br>
&gt; I&#39;m trying to find out if computation migration is something the d=
eveloper needs to worry about or if it&#39;s supposed to be hidden.<br>
&gt;<br>
&gt; I would like to use hadoop to take in a list of image paths in the hdf=
s and then have each task compress these large, raw images into something m=
uch smaller - say jpeg =A0files.<br>
&gt;<br>
&gt; Input: list of paths<br>
&gt; Output: compressed jpeg<br>
&gt;<br>
&gt; Since I don&#39;t really need a reduce task (I&#39;m more using hadoop=
 for its reliability and orchestration aspects), my mapper ought to just ta=
ke the list of image paths and then work on them. =A0As I understand it, ea=
ch image will likely be on multiple data nodes.<br>

&gt;<br>
&gt; My question is how will each mapper task &quot;migrate the computation=
&quot; to the data nodes? =A0I recall reading that the namenode is supposed=
 to deal with this. =A0Is it hidden from the developer? =A0Or as the develo=
per, do I need to discover where the data lies and then migrate the task to=
 that node? =A0Since my input is just a list of paths, it seems like the na=
menode couldn&#39;t really do this for me.<br>

&gt;<br>
&gt; Another question: Where can I find out more about this? =A0I&#39;ve lo=
oked up &quot;rack awareness&quot; and &quot;computation migration&quot; bu=
t haven&#39;t really found much code relating to either one - leading me to=
 believe I&#39;m not supposed to have to write code to deal with this.<br>

&gt;<br>
&gt; Anyway, could someone please help me out or set me straight on this?<b=
r>
&gt;<br>
&gt; Thanks,<br>
&gt; -Julian<br>
<br>
</div></div></blockquote></div><br></div></div>

--f46d042c6423edfdac04d73623c6--