hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alejandro Abdelnur (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
Date Wed, 03 Apr 2013 22:07:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621395#comment-13621395
] 

Alejandro Abdelnur commented on MAPREDUCE-4898:
-----------------------------------------------

While the patch looks good, we may have introduced a behavioral change in the {{setOutputPath()}}
method.

In Hadoop 1, {{setOutputPath()}} sets whatever path it is received in the conf, without any
modification.

In Hadoop 2, {{setOutputPath()}} makeQualified-s the received path and then sets it in the
conf.

This means that in Hadoop 1 a relative path will always resolved to the homedir while in Hadoop
2 a relative path will be resolved to the working dir (by default set to the homedir).

Can we please verify this hypothesis? This could be done by having a simple MR job where in
the submission we set (or not) the workingdirectory in the FS, and use a relative path for
the output dir. Then run the job once setting workingdir to /tmp, once not setting it at all.
Run this in Hadoop 1 and in Hadoop 2. Find out where the job output ends in all cases. If
different, we need to decide if we want to revert to Hadoop 1 behavior or not.

                
> FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with
MR1
> ------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4898
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>             Fix For: 2.0.5-beta
>
>         Attachments: MAPREDUCE-4898.patch
>
>
> In MR1, {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}}
throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it throws {{org.apache.hadoop.fs.FileAlreadyExistsException}}
instead, making them incompatible.  
> In MR1, {{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}} doesn't
throw any exceptions but now it throws an {{IOException}}, making them incompatible.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message