hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13055) Implement linkMergeSlash for ViewFileSystem
Date Fri, 09 Dec 2016 00:23:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733822#comment-15733822

Andrew Wang commented on HADOOP-13055:

Thanks for working on this Zhe and Manoj. I gave the patch a look, had some high-level comments
to go over first.

It looks like this is mostly config changes to allow mounting a filesystem on /. It doesn't
enable nested mounts though, so if you mount an FS on /, that's all you get. This doesn't
seem that useful, since the point of ViewFS is to combine multiple FS namespaces.

I interpret "linkMergeSlash" to mean having an FS mounted on /, but then allow the VFS mount
table to override that. Essentially, nested mounts, special cased to mounting over the "/"

Nested mounts are problematic though because we don't do path resolution fully on the client-side.
This surfaces for recursive delete; once we've deferred it to a mounted filesystem, it won't
pop back to VFS to get redirected to recurse through a nested mount. It won't behave like
"rm -rf". I haven't looked at the full FS interface to see what else is out there.

My question is why "/" is called out as a special case for nested mounts. It doesn't seem
easier to implement, and the semantic issues with recursive ops are still there. It's also
different from a merge mount, in that merging merges multiple FSs, while IIUC this is mounting
an FS *over* another FS.

[~sanjay.radia] could you provide any more historical perspective on the intent here?

HADOOP-8299 also provides a little hint from [~eli] in the description, where he refers to
the idea of a "default NN" for a mount table. This lets us ignore the semantic differences
with unix mounts, and simplifies the implementation as well (first try to resolve with the
VFS mount table, then fallback to the default FS).

> Implement linkMergeSlash for ViewFileSystem
> -------------------------------------------
>                 Key: HADOOP-13055
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13055
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs, viewfs
>            Reporter: Zhe Zhang
>            Assignee: Manoj Govindassamy
>         Attachments: HADOOP-13055.00.patch, HADOOP-13055.01.patch, HADOOP-13055.02.patch,
HADOOP-13055.03.patch, HADOOP-13055.04.patch
> In a multi-cluster environment it is sometimes useful to operate on the root / slash
directory of an HDFS cluster. E.g., list all top level directories. Quoting the comment in
> {code}
>  *   A special case of the merge mount is where mount table's root is merged
>  *   with the root (slash) of another file system:
>  *   <ul>
>  *   <li>    fs.viewfs.mounttable.default.linkMergeSlash=hdfs://nn99/
>  *   </ul>
>  *   In this cases the root of the mount table is merged with the root of
>  *            <b>hdfs://nn99/ </b> 
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message