hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14210) Directories are not listed recursively when fs.defaultFs is viewFs
Date Fri, 24 Mar 2017 01:21:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939527#comment-15939527

Andrew Wang commented on HADOOP-14210:

I don't have a fully educated opinion on this topic, but I normally look to Unixy behavior
to determine user expectations of HDFS. "ls -R /" lists everything. When I stat a mount point,
it's a directory, not a symlink.

Symlinks are currently not enabled in HDFS, so no downstream apps have ever encountered a
symlink. Again looking at Unix as a guide, applications still don't handle symlinks correctly
from a correctness or security POV after decades of existence. So, if we can roll out VFS
without requiring apps to understand symlinks, I'd be happy.

A lot of our apps use HDFS APIs like the shell or {{FileSystem#listFiles}}. We'd want these
to work the same way regardless of how the VFS is sharded, for transparency.

Can we dredge up any of the initial motivations from the VFS design docs or JIRA?

Given that existing FSs can already span to 100s of millions of files, clients need to handle
high scale no matter what. For the CLI ls command, I'm hoping we're using the iterator-based
listing methods, as well as providing a mode that doesn't try to align columns (which requires
buffering all the output).

> Directories are not listed recursively when fs.defaultFs is viewFs
> ------------------------------------------------------------------
>                 Key: HADOOP-14210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: viewfs
>    Affects Versions: 2.7.0
>            Reporter: Ajith S
>              Labels: viewfs
>         Attachments: HDFS-8413.patch
> Mount a cluster on client throught viewFs mount table
> Example:
> {quote}
>  <property>
>     <name>fs.defaultFS</name>
>     <value>viewfs:///</value>
>   </property>
>     <property>
>         <name>fs.viewfs.mounttable.default.link./nn1</name>
>         <value>hdfs://ns1/</value>  <!-- HA nameservice -->
>     </property>
>     <property>
>         <name>fs.viewfs.mounttable.default.link./user</name>
>         <value>hdfs://host-72:8020/</value>
>     </property>
>  <property>
> {quote}
> Try to list the files recursively *(hdfs dfs -ls -R / or hadoop fs -ls -R /)* only the
parent folders are listed.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message