hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5901) FileSystem.fixName() has unexpected behaviour
Date Tue, 15 Dec 2009 22:23:18 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790989#action_12790989

Aaron Kimball commented on HADOOP-5901:

Yes, changing the signature for {{FileSystem.setDefaultUri(Configuration, String)}} to throw
an IOException if it cannot create a proper URI from the String argument would be an incompatible

Digging into the changes necessary in the MapReduce codebase, it appears that  ({{MiniMRCluster.configureJobConf()}})
is the only use of FileSystem.setDefaultUri(). So that could be changed to itself throw IOException
easily enough. But that is called by {{MiniMRCluster.createJobConf()}} which is used in several
dozen places throughout the test codebase.

On the HDFS codebase side, it looks as though all the uses of setDefaultUri are in test cases
which can all be modified to throw more exceptions than they do now in a straightforward way;
or else in places where the URI is generated programmatically and we know that it's going
to be valid. (Hence we can swallow the IOException.)

So I think our options are as follows:

# Modify {{MiniMRCluster.createJobConf()}} to also throw IOException and modify all call-sites
to catch IOException.
# Early in this patch's development I was throwing an IllegalArgumentException (unchecked)
for this reason (preventing incompatibility). Konstantin S: Are you still pro-IOException
# Make the {{FileSystem.checkName()}} method public. The {{namenode}} argument in MiniMRCluster
which is used to format the URI is passed into MiniMRCluster's constructor. With a public
checkName, we could call {{checkName(namenode)}} in the MiniMRCluster constructor, which already
throws IOException, so clients already trap that. Then we could swallow the exception (maybe
with a log4j message) in {{createJobConf()}}, knowing that it should never be thrown since
we already verified the URI's sanity. This increases the size of the public static FileSystem

> FileSystem.fixName() has unexpected behaviour
> ---------------------------------------------
>                 Key: HADOOP-5901
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5901
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>            Assignee: Aaron Kimball
>            Priority: Minor
>             Fix For: 0.22.0
>         Attachments: HADOOP-5901.2.patch, HADOOP-5901.3.patch, HADOOP-5901.patch
> {{FileSystem.fixName()}} tries to patch up fs.default.name values, but I'm not sure it
helps that well. 
> Has it been warning about deprecated values for long enough for it to be turned off?

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message