hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9938) Sub classing of WebHDFS for the new file system implementation
Date Mon, 14 Mar 2016 00:32:33 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192586#comment-15192586

Chris Douglas commented on HDFS-9938:

As [discussed|https://issues.apache.org/jira/browse/HADOOP-12666?focusedCommentId=15186380&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15186380]
in the related JIRA, extending WebHDFS this way is undesirable. This connection will be [severed|https://issues.apache.org/jira/browse/HADOOP-12666?focusedCommentId=15189533&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15189533]
as the implementation evolves, implementing approach (3).

Locating part of the implementation in the same package (approach (1) in the description)
is not hygienic, but it's not without precedent (e.g., Mumak, yarn-server/yarn-server-common,
the fault injection tooling). For reviewers unfamiliar with this history, it's also clear
that future issues are the fault of the ADL implementation overriding package-private methods,
and this extension was not part of the WebHDFS design. If ADL impedes the development of WebHDFS,
then it would be permissible to delete it from the repository, as has been the fate of similar

Looking through the details of the patch (approach (2)): the visibility of the {{o.a.h.hdfs.web.\*Param}}
classes is inconsistent, so those changes are mostly neutral. However, relaxing visibility
in {{WebHdfsFileSystem}} is similar to approaches rejected elsewhere in HDFS. There must be
an API that defines a contract for extensions, and making methods public is insufficient.
HDFS-8154 is an example of how we _should_ define protocol compatibility for WebHDFS. Short
of that, refactoring WebHDFS to support overriding some internal details might allow multiple
implementations to reuse its machinery without ill effects, such as colliding configuration
variables (as [~mackrorysd] and others pointed out in HADOOP-12666).

A short-term implementation of approach (3) would copy the machinery of WebHDFS into the ADL
package. This seems strictly worse than the other approaches.

I favor approach (1), unless there's a cleaner design for extension. Given the review feedback
on HADOOP-12666, extending WebHDFS at all should be a stopgap measure.

Minor: the patch accidentally contains some pom modifications from HADOOP-12666. Also, would
{{protected}} be sufficient for most of the modifications to {{WebHdfsFileSystem}}?

> Sub classing of WebHDFS for the new file system implementation
> --------------------------------------------------------------
>                 Key: HDFS-9938
>                 URL: https://issues.apache.org/jira/browse/HDFS-9938
>             Project: Hadoop HDFS
>          Issue Type: Wish
>            Reporter: Vishwajeet Dusane
>         Attachments: HDFS-9938-001.patch
> This JIRA is to request opinion from the community, Can new file system use an extension
of WebHDFS file system. 
> Considering all the known limitation WebHDFS has over implementing a new client by extending
FileSystem class.
> Option we have is 
> 1. Use the namespace org.apache.hadoop.hdfs.web in new file system implementation to
access protected functionality from WebHdfsFileSystem.
> 2. Change the WebHdfs to support extensions
> 3. Suggestion on different approach like new client by sub classing FileSystem.

This message was sent by Atlassian JIRA

View raw message