hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10741) A lightweight WebHDFS client library
Date Tue, 24 Jun 2014 22:40:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14042795#comment-14042795
] 

Andrew Wang commented on HADOOP-10741:
--------------------------------------

Hey guys, I understand the need for something like this. The point you brought up about every
app having to reimplement their own FS on top of the REST API is a good one, and I've seen
it done before.

However, is there any reason why this library needs to be included in Hadoop itself? I'd like
to avoid introducing another FileSystem-like API, since it increases the breadth and duplication
of our user API (more maintenance and compat to worry about). We also already have a Java
WebHDFS client which uses FileSystem and works pretty well and has extensive testing.

Could this new client live on github or elsewhere as a separate project?

> A lightweight WebHDFS client library
> ------------------------------------
>
>                 Key: HADOOP-10741
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10741
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: tools
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Mohammad Kamrul Islam
>
> One of the motivations for creating WebHDFS is for applications connecting to HDFS from
outside the cluster.  In order to do so, users have to either
> # install Hadoop and use WebHdfsFileSsytem, or
> # develop their own client using the WebHDFS REST API.
> For #1, it is very difficult to manage and unnecessarily complicated for other applications
since Hadoop is not a lightweight library.  For #2, it is not easy to deal with security and
handle transient errors.
> Therefore, we propose adding a lightweight WebHDFS client as a separated library which
does not depend on Common and HDFS.  The client can be packaged as a standalone jar.  Other
applications simply add the jar to their classpath for using it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message