hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4044) Create symbolic links in HDFS
Date Tue, 14 Oct 2008 22:35:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639629#action_12639629

Doug Cutting commented on HADOOP-4044:

Sanjay> if someone happens to create a dir called "hdfs:" and valid path foo/bar under
it then the local file system will follow it and not return an error. Is this a concern?

Folks would have to use a double-slash in the symbolic link (which unix ignores) for Hadoop
to interpret it as an external link.  So folks would have to use single-slash when linking
to a directory whose name ends in colon to keep behavior consistent.  That doesn't seem like
a big imposition.

Sanjay> Hence, on a "file does not exist" error, the client side lib will have to check
if there are symlinks in the path.

Good point.  If we leverage Unix's automatic link following then we'd have to handle external
links by catching FileNotFoundException.  We could instead loop calling lstat(), following
links that are internal, and returning when either a link to a non file:// uri or an actual
file is found.  This would require native code, but, this is a unix-only feature and we already
provide native code for linux, so maybe that's acceptable.  Or we could call out to a shell,
but that would probably be too slow.  Note that Java 7 adds API support for symbolic links,
which will permit such looping logic, so that will provide a good way to implement this.

FileNotFoundException doesn't actually make this any easier, since it doesn't contain the
path we need, as far as I can tell.  We'd still have to loop in native code or a shell script
to find the external link, no?  So maybe the native code route is best?  We could implement
just this bit of the Java 7 API, so when that arrives we can use it without changing much?

> Create symbolic links in HDFS
> -----------------------------
>                 Key: HADOOP-4044
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4044
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: HADOOP-4044-strawman.patch, symLink1.patch, symLink1.patch, symLink4.patch,
symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch
> HDFS should support symbolic links. A symbolic link is a special type of file that contains
a reference to another file or directory in the form of an absolute or relative path and that
affects pathname resolution. Programs which read or write to files named by a symbolic link
will behave as if operating directly on the target file. However, archiving utilities can
handle symbolic links specially and manipulate them directly.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message