hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakob Homan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10741) A lightweight WebHDFS client library
Date Thu, 02 Oct 2014 04:03:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156055#comment-14156055

Jakob Homan commented on HADOOP-10741:

The REST protocol exposed by the NameNode and consumed by the WebHDFS FileSystem implementation
is extremely valuable. It's the easiest point of access for non-JVM clients.  A non-oah.FileSystem
consumer implementation will exist, whether it's in Hadoop proper or out in github limbo.
 It'd be better to have the library here to avoid bitrot of the client, drift in implementations
and duplicated work.  Going further (and in a future JIRA), we should look at codifying the
server-side REST protocol through something like [RAML|http://raml.org/] or [Swagger|https://helloreverb.com/developers/swagger]
so that it's easy for other systems to offer access through it, in the same way that other
implementations of oah.FileSystem make those systems accessible.

> A lightweight WebHDFS client library
> ------------------------------------
>                 Key: HADOOP-10741
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10741
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: tools
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Mohammad Kamrul Islam
> One of the motivations for creating WebHDFS is for applications connecting to HDFS from
outside the cluster.  In order to do so, users have to either
> # install Hadoop and use WebHdfsFileSsytem, or
> # develop their own client using the WebHDFS REST API.
> For #1, it is very difficult to manage and unnecessarily complicated for other applications
since Hadoop is not a lightweight library.  For #2, it is not easy to deal with security and
handle transient errors.
> Therefore, we propose adding a lightweight WebHDFS client as a separated library which
does not depend on Common and HDFS.  The client can be packaged as a standalone jar.  Other
applications simply add the jar to their classpath for using it.

This message was sent by Atlassian JIRA

View raw message