hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
Date Thu, 03 Oct 2013 17:54:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785401#comment-13785401
] 

Colin Patrick McCabe commented on HADOOP-9984:
----------------------------------------------

Hi all,

We're having a webex about this at 3pm (PST).  It should be about an hour.  If you're interested,
the number is 1-650-479-3208.  Access code is 626 743 933.

bq. suresh said: I also want to understand why HADOOP-9972 has to be incompatible.

I don't think HADOOP-9972 needs to be incompatible.  It's just adding new APIs, not changing
existing.

bq. daryn asked: To help expedite the discussion, would you please enumerate exactly how the
current patch is returning paths (resolved vs. unresolved) in the file statuses returned from
getFileStatus, listStatus, globStatus, and the link variants?

Nothing has changed with regards to paths.  They're still always returned resolved.  That's
the way symlinks have been handled since they were first added to Hadoop and this patch doesn't
affect that.  The design makes this necessary, unless you want to do multiple link resolution
RPCs every time you use a path, which is not scalable. 

bq. Steve said: Due to the way the Hadoop FS interface is designed now, \[glusterfs\] can
create a single Hadoop 2.x plugin for glusterfs that is backwards compatible, so we just have
one.

Thanks for the data point.  If we choose to keep compatibility, you will have more work to
do to implement symlinks, but you'll be able to continue supporting pre-GA versions with the
same code.  I don't know desirable that is, since presumably people should be moving to GA
or releases based off of it soon.

> FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-9984
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9984
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.1.0-beta
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>            Priority: Blocker
>         Attachments: HADOOP-9984.001.patch, HADOOP-9984.003.patch, HADOOP-9984.005.patch,
HADOOP-9984.007.patch, HADOOP-9984.009.patch, HADOOP-9984.010.patch, HADOOP-9984.011.patch,
HADOOP-9984.012.patch, HADOOP-9984.013.patch, HADOOP-9984.014.patch
>
>
> During the process of adding symlink support to FileSystem, we realized that many existing
HDFS clients would be broken by listStatus and globStatus returning symlinks.  One example
is applications that assume that !FileStatus#isFile implies that the inode is a directory.
 As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning
resolved paths.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message