hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rémy SAISSY (JIRA) <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5264) FileAlreadyExistsException is assumed to be thrown by FileSystem#mkdirs or FileContext#mkdir in the codebase
Date Fri, 07 Jun 2013 21:48:20 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rémy SAISSY updated MAPREDUCE-5264:
-----------------------------------

    Target Version/s: 3.0.0, 2.1.0-beta
              Status: Patch Available  (was: Open)

Hi,
in JobHistoryEverntHandler and HistoryFileManager, the FileAlreadyExistsException was only
tracked for Logging purpose.

YarnChild.java::configureLocalDirs() tracked the FileAlreadyExistsException to determine which
task should open the directory for write and which one for read.

<code>
      workDir = lDirAlloc.getLocalPathForWrite("work", job);
      FileSystem lfs = FileSystem.getLocal(job).getRaw();
      boolean madeDir = false;
      try {
        madeDir = lfs.mkdirs(workDir);
      } catch (FileAlreadyExistsException e) {
        madeDir = true;
        workDir = lDirAlloc.getLocalPathToRead("work", job);
}
<code>

However the method's goal according to its documentation is to create the mapred-local output
directory. Shouldn't it be writable anyway?
I have removed the try catch and the read only open in the patch.

No need to modify or add any test case.
                
> FileAlreadyExistsException is assumed to be thrown by FileSystem#mkdirs or FileContext#mkdir
in the codebase
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5264
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5264
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 3.0.0, 2.1.0-beta
>            Reporter: Rémy SAISSY
>
> According to https://issues.apache.org/jira/browse/HADOOP-9438,
> FileSystem#mkdirs and FileContext#mkdir do not throw FileAlreadyExistsException if the
directory already exist.
> Some places in the mapreduce codebase assumes FileSystem#mkdirs or FileContext#mkdir
throw FileAlreadyExistsException.
> At least the following files are concerned:
>  - YarnChild.java
>  - JobHistoryEverntHandler.java
>  - HistoryFileManager.java
> It would be good to re-review and patch this if needed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message