hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
Date Thu, 12 Jun 2014 18:59:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14029620#comment-14029620

Chris Nauroth commented on HADOOP-10663:

I'm not in favor of committing this patch.  This is a continuation of comments that I left
on MAPREDUCE-5912 here:


The issue raised in MAPREDUCE-5912 is that a change was made that started looking for map
outputs on the default file system instead of the local file system.  This causes problems
on Windows if HDFS is the default file system, because the path will have a colon due to the
drive specifier, and a colon isn't a valid character in an HDFS file name.  As I said in my
MAPREDUCE-5912 comment, I don't think that change was correct, even if it appears to be working
in non-Windows environments.

For the patch here, imagine a situation where the default file system is neither the local
file system nor HDFS, and that file system implementation does allow colons in file names.
 With this patch, a {{Path}} with no scheme and a path that "looks like" a Windows path with
a drive spec can get misrouted to local file system instead of the default file system.  I
admit this is somewhat contrived, but it sure would create confusion if anyone ran into it.

I'd like to resolve this as won't fix.  [~rusanu] and [~chris.douglas], does this sound OK?

> Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
> ---------------------------------------------------------------------------------------
>                 Key: HADOOP-10663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10663
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: Remus Rusanu
>            Assignee: Remus Rusanu
>         Attachments: HADOOP-10663.1.patch
> While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking
a Windows specific Path to get its file system should not return the default FS, but the localFS.
Specifically, I believe that Path.initialize should identify such names as belonging to the
'file://' scheme (when null) and so create appropriate URIs.
> I do not have the know-how to evaluate the bigger impact of such a change.

This message was sent by Atlassian JIRA

View raw message