hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8409) Address Hadoop path related issues on Windows
Date Wed, 23 May 2012 20:21:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281875#comment-13281875

Daryn Sharp commented on HADOOP-8409:

bq. Aksing users to enter input paths in form "c:/some/path" does not seem like the right
thing to do.  Please let me know if you agree with me here. 
Unfortunately I disagree... 

HADOOP-8139 has a long discussion, but the short summary is:  Hadoop uses URIs, so the hadoop
veterans decided the hacky support for \ needs to be removed.  Although "dir/file" might look
like a native path, it's still a URI being prepending with the default fs URI.  I spun off
HADOOP-8164 to fix the meta-char quoting issue caused by the windows hack -- but bug persists
on windows!

HADOOP-8139 was linked to the window umbrella jira (HADOOP-8079) over a year ago, with the
reporter Alexander quoted as "We are only using backslashes () for escaping meta-characters
and not as alternative path-separators. So, things should work in a uniform manner."

Of note, it looks like for at least the past 6 years, MS has said URIs using \\ is deprecated?

bq.  Actually, this does not work for paths that are symlinks.  For example, new Path("/some/path#symlink")
will encode the "#" character internally, so we lose the symlink behavior.

I'm sorry, but you've completely lost me on how URI fragments are related to symlinks?  Is
this a windows convention?
> Address Hadoop path related issues on Windows
> ---------------------------------------------
>                 Key: HADOOP-8409
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8409
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs, test, util
>    Affects Versions: 1.0.0
>            Reporter: Ivan Mitic
>            Assignee: Ivan Mitic
>         Attachments: HADOOP-8409-branch-1-win.patch
>   Original Estimate: 168h
>  Remaining Estimate: 168h
> There are multiple places in prod and test code where Windows paths are not handled properly.
From a high level this could be summarized with:
> 1. Windows paths are not necessarily valid DFS paths (while Unix paths are)
> 2. Windows paths are not necessarily valid URIs (while Unix paths are)
> #1 causes a number of tests to fail because they implicitly assume that local paths are
valid DFS paths (by extracting the DFS test path from for example "test.build.data" property)
> #2 causes issues when URIs are directly created on path strings passed in by the user

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message