hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: relative symbolic links in HDFS
Date Mon, 31 Oct 2011 18:45:29 GMT
On Mon, Oct 31, 2011 at 9:27 AM, Charles Baker <cbaker@sdl.com> wrote:
> Hey guys. Thanks for the replies. Fully qualified symbolic links are
> problematic in that when we wish to restore a directory structure containing
> symlinks from HDFS to local filesystem, the relativity is lost. For instance:
> /user/cbaker/foo/
>                link1 -> ../../cbaker
> The current behavior of getFileLinkStatus() results in a path for link 1
> being:
> /user/cbaker
> Not:
> ../../cbaker
> Also, some symlinks may point to non-existent locations within HDFS which
> only have relevance to the local filesystem. This appears as though it could
> (though I haven't tested yet) result in an exception when the attempt is made
> to qualify it. If I get a chance, I'll try it out later today.
> FileContext.getLinkTarget() doesn't work for this case since it returns only
> the final component of the target, not the complete relative path.

Really?  FC#getLinkTarget should return the target verbatim, as
specified by the user when creating the link:

Eg see test testCreateLinkToDotDotPrefix:
 fc.createSymlink(new Path("../file"), link, false);
 assertEquals(new Path("../file"), fc.getLinkTarget(link));

> But even
> if it did return the relative path, it seems counter-intuitive to me. I agree
> with Daryn and expect the behavior of getFileLinkStatus() to return the
> symlink as is and not presume that I wanted it qualified. If I wanted a
> qualified path for a symlink, I would expect to call Path.makeQualified() to
> do so.

It does this because getFileStatus always returns fully qualified
paths in HDFS, and we don't make to make callers check the type and
care about the method that was used to obtain the FileStatus, eg to
know whether it contains a fully qualified path or not.

I think the original rationale for while FileStatus objects always
have fully qualified paths is so they can be passed around w/o callers
having to do future work to access them ie didn't want to disassociate
the path from the file system it exists on. Note that in Hadoop
"Paths" are actually URIs, vs file system paths (a subset of URIs).

Regardless of the rationale, changing getFileStatus to return objects
w/o fully qualified paths would break compatibility with a lot of
existing programs. It would also hinder people porting to FileContext
which tries to be consistent with FileSystem.

Would a new method on FileStatus or Path that returns the unqualified
version of the path (ie w/o the scheme and authority, and w/o
resolving relative paths relative to the FileContext) work?  Ie the
FileStatus could return the contents of the HdfsFileStatus w/o making
it fully qualified.


View raw message