hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Ryaboy <dvrya...@gmail.com>
Subject Re: purpose of JobTracker copying job.jar
Date Tue, 10 May 2011 10:41:12 GMT
Looking around 0.22, looks like this behavior is gone there, it's not
copying the jar. The only copyToLocal call it makes is to get the job
file (not the jar file).  0.20 did both.

Does anyone remember why 0.20 needed the jar?

-D

On Mon, May 9, 2011 at 12:55 AM, Raghu Angadi <rangadi@apache.org> wrote:
> On Wed, Apr 27, 2011 at 11:20 AM, Arun C Murthy <acm@yahoo-inc.com> wrote:
>>
>> So that it can instruct TTs to copy it from the system-dir for actually
>> running jobs..
>
> Thanks Arun.
> Task trackers copy the job jar from HDFS (placed by client).
> JobTracker copying the same file to its local filesystem seems to be
> unnecessary.. This could have very noticeable effects on the central server.
> I commented out JobInProgress.java:266 on 0.20 trunk, and my test jobs run
> fine. I will try to double check what I am missing.
> Raghu.
>
>>
>> hope that helps,
>> Arun
>>
>> On Apr 26, 2011, at 11:57 AM, Raghu Angadi wrote:
>>
>>>
>>> Job tracker copies job jar from mapred system directory on HDFS to its
>>> local file system ( ${mapred.local.dir}/jobTracker ).
>>>
>>> What is the purpose of copying this file? It does not run any user code.
>>>
>>> thanks,
>>> Raghu.
>>>
>>
>
>

Mime
View raw message