hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mukul Kumar Singh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11575) Supporting HDFS NFS gateway with Federated HDFS
Date Fri, 16 Jun 2017 15:16:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16052022#comment-16052022
] 

Mukul Kumar Singh commented on HDFS-11575:
------------------------------------------

Thanks for the review [~jnp], Please find my replies inline.

Can we avoid calling fs.resolvePath(..)? This makes a server call and needs HDFS to be up
when NFS is being deployed. Not too bad, but decoupling would be better, as it has been without
this patch. One possibility is to make path qualified using file system uri. If creating a
URI is lightweight, we may not need to store path to URI mapping. The mount function already
makes a call to Namenode, URI creation could also be moved there.
bq. This step is needed because entries in the mountable map needs to be parsed an resolved.
This is will perform the step of parsing the {{InodeTree.ResolveResult}}.

To construct DFSClient, we could use this constructor DFSClient(URI nameNodeUri, Configuration
conf), will save a step. The constructor should take care of HA case.
bq. Done

I am wondering why do we need 'hostMapAddress', since we are caching DFSClient itself, the
lookups to this map would be rare, in that case a DNS resolution is not that bad.
bq. True, access to this map will be rare, however this map is needed when a cached DFSClient
is evicted from the cache. This map will be used to map a name node id to its corresponding
URI.

{{ FileSystem[] childFileSystems = fs.getChildFileSystems()}} This line will get the children
file systems of the target as well. It might be better to get the target file systems from
the mount points. We may not need it altogether if hostMapAddress is removed.
bq. Great suggestion, done :)

TestViewfsWithNfs3: Please add a test for rename to check for unsupported error.
bq. Done

Add a test to start NFS service with viewfs over a non-hdfs file system. It is ok to add it
in a follow up jira.
bq. Will add this as part of follow up jira

> Supporting HDFS NFS gateway with Federated HDFS
> -----------------------------------------------
>
>                 Key: HDFS-11575
>                 URL: https://issues.apache.org/jira/browse/HDFS-11575
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: nfs
>            Reporter: Mukul Kumar Singh
>            Assignee: Mukul Kumar Singh
>         Attachments: HDFS-11575.001.patch, HDFS-11575.002.patch, HDFS-11575.003.patch,
HDFS-11575.004.patch, HDFS-11575.005.patch, SupportingNFSwithFederatedHDFS.pdf
>
>
> Currently HDFS NFS gateway only supports HDFS as the underlying filesystem.
> Federated HDFS with ViewFS helps in improving the scalability of the name nodes. However
NFS is not supported with ViewFS.
> With this change, ViewFS using HDFS as the underlying filesystem can be exported using
NFS. ViewFS mount table will be used to determine the exports which needs to be supported.
> Some important points
> 1) This patch only supports HDFS as the underlying filesystem for ViewFS.
> 2) This patch add support to add more than one export point in the NFS gateway
> 3) Root filesystem of the ViewFS will not be mountable for NFS gateway with ViewFS,
>    however this will not be the case for NFS gateway with HDFS
> 4) A filehandle now apart from the field will also contain an identifier to identify
the name node, this will be used to map to correct name node for file operations.
> Please find the attached pdf document which helps in explaining the design and the solution.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message