Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
From: Robert Evans <evans@yahoo-inc.com>
To: "mapreduce-user@hadoop.apache.org" <mapreduce-user@hadoop.apache.org>
Date: Fri, 27 May 2011 07:54:00 -0700
Subject: Re: How does Map and Reduce class are sent to remote node by hadoop
 ??
Thread-Topic: How does Map and Reduce class are sent to remote node by
 hadoop ??
Thread-Index: AcwceOR5XJdVpaaAQWuhu1VVOtH5ewABQT6o
Message-ID: <CA052538.23E52%evans@yahoo-inc.com>
In-Reply-To: <BANLkTinzaiKuj1uyVaD_JK1zUNOqQAsG7Q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_CA05253823E52evansyahooinccom_"
MIME-Version: 1.0

--_000_CA05253823E52evansyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Francesco,

The mapreduce client will create a jar called job.jar and place it in HDFS =
in a staging directory.  This is the jar that you specified to your job con=
f, or I believe that it tries to guess the jar based off of the Mapper clas=
s and the Reducer class but I am not sure of that.  Once the job tracker ha=
s told a TaskTracker to run a given job the TaskTracker will download the j=
ar, and then fork off a new JVM to execute the Mapper or Reducer.  If you j=
ar has dependencies then these usually have to be shipped with it as part o=
f the cache archive interface.

--Bobby Evans

On 5/27/11 9:16 AM, "Francesco De Luca" <f.deluca86@gmail.com> wrote:

Anyone knows the mechanism that hadoop use to load Map and Reduce class on =
the remote node
where the JobTracker submit the tasks?

In particular, how can hadoop retrieves the .class files ?

Thanks


--_000_CA05253823E52evansyahooinccom_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<HTML>
<HEAD>
<TITLE>Re: How does Map and Reduce class are sent to remote node by hadoop =
??</TITLE>
</HEAD>
<BODY>
<FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:=
11pt'>Francesco,<BR>
<BR>
The mapreduce client will create a jar called job.jar and place it in HDFS =
in a staging directory. &nbsp;This is the jar that you specified to your jo=
b conf, or I believe that it tries to guess the jar based off of the Mapper=
 class and the Reducer class but I am not sure of that. &nbsp;Once the job =
tracker has told a TaskTracker to run a given job the TaskTracker will down=
load the jar, and then fork off a new JVM to execute the Mapper or Reducer.=
 &nbsp;If you jar has dependencies then these usually have to be shipped wi=
th it as part of the cache archive interface.<BR>
<BR>
--Bobby Evans<BR>
<BR>
On 5/27/11 9:16 AM, &quot;Francesco De Luca&quot; &lt;<a href=3D"f.deluca86=
@gmail.com">f.deluca86@gmail.com</a>&gt; wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"=
><SPAN STYLE=3D'font-size:11pt'>Anyone knows the mechanism that hadoop use =
to load Map and Reduce class on the remote node<BR>
where the JobTracker submit the tasks?<BR>
<BR>
In particular, how can hadoop retrieves the .class files ?<BR>
<BR>
Thanks<BR>
<BR>
</SPAN></FONT></BLOCKQUOTE>
</BODY>
</HTML>


--_000_CA05253823E52evansyahooinccom_--