Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Wed, 24 Jul 2013 22:53:50 +0000 (UTC)
From: "Colin Patrick McCabe (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12659221.1374534186331.107494.1374706430723@arcas>
In-Reply-To: <JIRA.12659221.1374534186331@arcas>
References: <JIRA.12659221.1374534186331@arcas>
Subject: [jira] [Commented] (HADOOP-9761) ViewFileSystem#rename fails when
 using DistributedFileSystem
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-9761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718969#comment-13718969 ] 

Colin Patrick McCabe commented on HADOOP-9761:
----------------------------------------------

{code}
    // Assumes path belongs to this FileSystem.
    // Callers are responsible for checking this
    FileSystem fs = filesys;
{code}

I don't see where you are actually checking that the path belongs to the FileSystem in most of the callers.

For example, back before symlinks support in FileSystem, code like this:
{code}
FileSystem fs = FileSystem.get(hdfs...);
fs.open("file:///boguspath");
{code}

Would generate a stacktrace like this:
{code}
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: file:/boguspath, expected: hdfs://localhost:6000
        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:590)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:170)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:246)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:79)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:711)
{code}

After the symlinks support, it generated this:
{code}
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS: file:/tmp/t, expected: hdfs://localhost:6000
        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:636)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:181)
        at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:91)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:290)
        at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:286)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:78)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:286)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:757)
{code}

It seems like this change would simply pass the incorrect path through to the FileSystem, which seems wrong.

treating unqualified (i.e., "schemeless") Paths as owned by the current FileSystem should not require disabling the checks for Paths which *do* have a wrong scheme.
                
> ViewFileSystem#rename fails when using DistributedFileSystem
> ------------------------------------------------------------
>
>                 Key: HADOOP-9761
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9761
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: viewfs
>    Affects Versions: 3.0.0, 2.1.0-beta
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>         Attachments: hadoop-9761-1.patch
>
>
> ViewFileSystem currently passes unqualified paths (no scheme or authority) to underlying FileSystems when doing a rename. DistributedFileSystem symlink support added in HADOOP-9418 needs to qualify and check rename sources and destinations since cross-filesystem renames aren't supported, so this breaks in the following way
> - Default FS URI is configured to viewfs://<viewfs>
> - When doing a rename, ViewFileSystem checks to make sure both src and dst FileSystems are the same (which they are, both in same DFS), and then calls DistributedFileSystem#rename with unqualified "remainder" paths
> - Since these paths are unqualified, DFS qualifies them with the default FS to check that it can do the rename. This turns it into viewfs://<viewfs>/<path>
> - Since viewfs://<viewfs> is not the DFS's URI, DFS errors out the rename.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira