hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mikhail Erofeev (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12292) Federation: Support viewfs:// schema path for DfsAdmin commands
Date Tue, 22 Aug 2017 20:12:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16137312#comment-16137312

Mikhail Erofeev commented on HDFS-12292:

Hey [~msingh], thank you. 
1) The logic is as follows: if it is not a DFS, we try to resolve it to another one, it is
possible for ViewFs/HarFs/FilterFs. If it is still not a DFS, then we raise an exception.
2) The first check compares just some Strings. So PathData src, in theory, can be different
only in a slash or an absent schema (it is not true now, as the only call happens after expandAsGlob(),
and it normalizes paths for us). So if the paths check fails, we still can compare the source
fs and the resolved one, and if they are the same, we can skip FileStatus resolve, I think.
But it is just a premature optimization and I don't mind to remove it.
3) There is a contract in fs.resolvePath() that the returned path is fully qualified. 

> Federation: Support viewfs:// schema path for DfsAdmin commands
> ---------------------------------------------------------------
>                 Key: HDFS-12292
>                 URL: https://issues.apache.org/jira/browse/HDFS-12292
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: federation
>            Reporter: Mikhail Erofeev
>            Assignee: Mikhail Erofeev
>         Attachments: HDFS-12292-002.patch, HDFS-12292-003.patch, HDFS-12292-004.patch,
> Motivation:
> As of now, clients need to specify a nameservice when a cluster is federated, otherwise,
the exception is fired:
> {code}
> hdfs dfsadmin -setQuota 10 viewfs://vfs-root/user/uname
> setQuota: FileSystem viewfs://vfs-root/ is not an HDFS file system
> # with fs.defaultFS = viewfs://vfs-root/
> hdfs dfsadmin -setQuota 10 vfs-root/user/uname
> setQuota: FileSystem viewfs://vfs-root/ is not an HDFS file system
> # works fine thanks to https://issues.apache.org/jira/browse/HDFS-11432
> hdfs dfsadmin -setQuota 10 hdfs://users-fs/user/uname
> {code}
> This creates inconvenience, inability to rely on fs.defaultFS and forces to create client-side
mappings for management scripts
> Implementation:
> PathData that is passed to commands should be resolved to its actual FileSystem
> Result:
> ViewFS will be resolved to the actual HDFS file system

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message