hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10663) Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
Date Thu, 12 Jun 2014 19:57:03 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14029703#comment-14029703

Chris Nauroth commented on HADOOP-10663:

bq. That code should work if {{mapOutput}} is a fully qualified {{Path}}, right?

Yes, that's correct.  You mentioned that MAPREDUCE-5269 includes changes to guarantee the
{{Path}} instances are fully qualified.  Thanks for pointing that out.  The problem is that
MAPREDUCE-5269 isn't committed yet, and it's a larger patch that will take more time to review,
so I'm going to proceed with committing MAPREDUCE-5912 for a short-term fix.

bq. Any heuristic that infers the FS from name can be defeated by a carefully constructed
future FS  In other words scheme should be mandatory

In general, yes, I agree.  I prefer using a fully-qualified {{Path}}, particularly around
service or process boundaries where the {{Configuration}} instances might be different.  Even
better would be a way to statically type-check relative vs. fully-qualified, but this is a
long-standing limitation of the {{Path}} class and the underlying {{URI}} that it wraps.

However, we've also historically had the behavior that if the {{Path}} doesn't have a scheme,
then it gets mapped to the default {{FileSystem}} based on configuration.  Like you said,
it's imperfect, but it's sometimes convenient, and we need to maintain that behavior for backwards-compatibility.
 The patch here would create a risk of bypassing that logic, which is an expected part of
the contract.

> Path.getFileSystem should identify Windows drive (c:) like files and return the localFS
> ---------------------------------------------------------------------------------------
>                 Key: HADOOP-10663
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10663
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: Remus Rusanu
>            Assignee: Remus Rusanu
>         Attachments: HADOOP-10663.1.patch
> While investigating the problem with MAPREDUCE-5912 I considered that the fact that asking
a Windows specific Path to get its file system should not return the default FS, but the localFS.
Specifically, I believe that Path.initialize should identify such names as belonging to the
'file://' scheme (when null) and so create appropriate URIs.
> I do not have the know-how to evaluate the bigger impact of such a change.

This message was sent by Atlassian JIRA

View raw message