hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-52) mapred input and output dirs must be absolute
Date Wed, 22 Mar 2006 06:55:17 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-52?page=all ]

Owen O'Malley updated HADOOP-52:

    Attachment: cwd.patch

Here is a patch that fixes the problem. 

It does:
   1. It adds {set,get}WorkingDirectory to FileSystem.
   2. It implements them in both LocalFileSystem and DFS.
   3. The LocalFileSystem implementation both sets the System property user.dir and does an
        conversion to absolute filenames at the API.
   4. Added new junit test cases to test the WorkingDirectory functionality.
   5. Added a utility class in the test directory to create a single-process DFS cluster for
junit tests.
   6. Added the user name into the JobConf.
   7. Added the user name into the JobProfile.
   8. Added the user name into the webapp, so you can see who ran the job.
   9. Added the working directory in the default file system to the JobConf.
   10. Set the job's working directory before starting the user's Map or Reduce code. (The
input splitter is given an absolute pathname
          for the input directory, but the working directory is not set, since it is done
in the context of the JobTracker.)
   11. Changed the format of the percentage complete in the webapp to be ##0.00 so that you
don't get 16 digits of meaningless precision
          about your job status.

> mapred input and output dirs must be absolute
> ---------------------------------------------
>          Key: HADOOP-52
>          URL: http://issues.apache.org/jira/browse/HADOOP-52
>      Project: Hadoop
>         Type: Bug
>   Components: mapred
>     Versions: 0.1
>     Reporter: Doug Cutting
>     Assignee: Owen O'Malley
>      Fix For: 0.1
>  Attachments: cwd.patch
> DFS converts relative pathnames to be under /user/$USER.  But MapReduce jobs may be submitted
by a different user than is running the jobtracker and tasktracker.  Thus relative paths must
be resolved before a job is submitted, so that only absolute paths are seen on the job tracker
and tasktracker.  I think the simplest way to fix this is to make JobConf.setInputDir(), setOutputDir(),
etc. resolve relative pathnames. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message