hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subramaniam Krishnan (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-3135) if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value job submission fails
Date Wed, 16 Apr 2008 06:52:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12589438#action_12589438
] 

subru edited comment on HADOOP-3135 at 4/15/08 11:50 PM:
------------------------------------------------------------------------

I have updated the patch with the following changes:

1) Went through the call hierarchy of FileSystem.get(conf) with Amareshwari's help and updated
a few instances to make sure they are correct(in my limited knowledge). 
2) Deprecated JobConf.getSystemDir. Replaced all references to JobConf.getSystemDir in our
code and change them to use JobClient.getSystemDir.
3) Add caching for the value of the system directory in JobClient.getSystemDir.

I already have added a test case that checks if a Job run successfully in spite of different
system directory configuration between Job Client & Job Tracker. I couldn't set the system
directory to a non-default file system as the MiniMRCluster doesn't support it currently.
Have raised a jira for the same -  [HADOOP-3244]

I haven't changed the imports as they were giving me warnings & are much better organized
now.

      was (Author: subru):
    I have updated the patch with the following changes:

1) Went through the call hierarchy of FileSystem.get(conf) with Amareshwari's help and updated
a few instances to make sure they are correct(in my limited knowledge). 
2) Deprecated JobConf.getSystemDir. Replaced all references to JobConf.getSystemDir in our
code and change them to use JobClient.getSystemDir.
3) Add caching for the value of the system directory in JobClient.getSystemDir.

I already have added a test case that checks if a Job run successfully in spite of different
system directory configuration between Job Client & Job Tracker. I couldn't set the system
directory to a non-default file system as the MiniMRCluster doesn't support it currently.
Have raised a jira for the same -  [HADOOP-3244][https://issues.apache.org/jira/browse/HADOOP-3244]

I haven't changed the imports as they were giving me warnings & are much better organized
now.
  
> if the 'mapred.system.dir' in the client jobconf is different from the JobTracker's value
job submission fails
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-3135
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3135
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.16.1
>         Environment: all
>            Reporter: Alejandro Abdelnur
>            Assignee: Subramaniam Krishnan
>            Priority: Critical
>             Fix For: 0.18.0
>
>         Attachments: patch-3135-v3.txt, patch-3135-v4.txt, patch-3135-v5.txt
>
>
> Until Hadoop 0.13 or so, at submission time the full path of the job.xml and all supporting
files in DFS was given by the client to the jobtracker.
> Since 0.15 onwards (we did not test 0.14) the jobclient is obtaining the job ID from
the jobtracker and creating the directory for all the supporting files using the a system-dir
computed from the local jobconf.
> Line 696-7 in the JobClient:
>     String jobId = jobSubmitClient.getNewJobId();
>     Path submitJobDir = new Path(job.getSystemDir(), jobId);
> This makes submissions to fail when the value of the 'mapred.system.dir' on the client
is different from the one in the JobTracker.
> A simple way o fixing this would be to introduce a new method in the JobSubmissionProtocol
'getSystemDir()' that would return the jobtracker system dir and use that dir for uploading
all the files on submission.
> ----
> For the future: A more comprehensive way of this doing would to obtain a base jobConf
from the jobtracker, carrying final information for each element and the construct the job.xml
on the client using the final semantics. And, in this case the 'mapred.system.dir' property
should be set as final in the jobtracker. As there may be some configuration properties that
are sensitive and for security reasons should not be exposed to the clients a new flag 'private'
could be introduced and only properties that don't have the 'private' flag would be send over
from the jobtracker to the jobclient for job.xml resolution.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message