hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakob Homan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8154) Extract WebHDFS protocol out as a specification to allow easier clients and servers
Date Wed, 15 Apr 2015 21:24:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497038#comment-14497038
] 

Jakob Homan commented on HDFS-8154:
-----------------------------------

Currently WebHDFS exists as a [human-readable specification|https://hadoop.apache.org/docs/r2.5.1/hadoop-project-dist/hadoop-hdfs/WebHDFS.html]
and a single implementation, that provided by the NameNode and its DataNodes.  We should extract
WebHDFS out as a specification, which would allow easier implementation of both WebHDFS-backed
servers and clients.  Additionally, this would make it easier to generate documentation and
verify correctness.
The current human-readable spec makes it possible to implement WebHDFS clients (for example,
[Perl|http://search.cpan.org/~afaris/Apache-Hadoop-WebHDFS-0.04/lib/Apache/Hadoop/WebHDFS.pm],
[Python|https://pypi.python.org/pypi/pywebhdfs], [.NET|https://hadoopsdk.codeplex.com/wikipage?title=WebHDFS%20Client]
, [non-Hadoop backed JVM|https://issues.apache.org/jira/browse/HADOOP-10741], etc.).  However,
each client must be built by someone parsing out that spec and writing up their own client
implementation from scratch.

There are frameworks, such as [Swagger|http://swagger.io/] and [RAML|http://raml.org/] that
allow one to define a REST interface and then create documentation, generate client stubs
and built tests against that framework.
In addition to clients, an more programmatic WebHDFS specification would allow other backend
systems to more easily implement the WebHDFS interface.  Any Hadoop application would then
be able to access that back end through the oah.WebHDFSFileSystem or through one of the non-JVM
clients described above.

This JIRA will cover specifying the WebHDFS spec in a framework like Swaggger or RAML, switching
the WebHDFS documentation to be built from this so as to be authoritative, and verifying that
the current implementation provided by the namenode and datanodes comports with this specification.

> Extract WebHDFS protocol out as a specification to allow easier clients and servers
> -----------------------------------------------------------------------------------
>
>                 Key: HDFS-8154
>                 URL: https://issues.apache.org/jira/browse/HDFS-8154
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: webhdfs
>            Reporter: Jakob Homan
>            Assignee: Jakob Homan
>
> WebHDFS would be more useful if there were a programmatic description of its interface,
which would allow one to more easily create servers and clients.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message