hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mohammad Kamrul Islam (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10741) A lightweight WebHDFS client library
Date Tue, 24 Jun 2014 02:06:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041603#comment-14041603

Mohammad Kamrul Islam commented on HADOOP-10741:

[~tucu00] thanks for the comments. 
One of the key requirement is to provide a "light-weight" library that is independent of hadoop
core. The independence of core is required because any update to hadoop service means the
upgrade of application  as well (which might need to go through the full qualification life
cycle).  This is the pain for a non-hadoop application that runs outside Hadoop cluster and
occasionally retrieves file from Hadoop.

I agree to reuse the hadoop-auth. But dependent of hadoop core jar will miss the key requirement.

I also agree that some of this (scaled down version) needs to be re-implemented. If you have
some idea how to achieve the both : not to re-implement and not to depend on hadoop core library,
it will be great.

In short, this requirement focuses on the application running outside hadoop but needs to
occasionally get/put the data from/into hadoop.


> A lightweight WebHDFS client library
> ------------------------------------
>                 Key: HADOOP-10741
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10741
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: tools
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Mohammad Kamrul Islam
> One of the motivations for creating WebHDFS is for applications connecting to HDFS from
outside the cluster.  In order to do so, users have to either
> # install Hadoop and use WebHdfsFileSsytem, or
> # develop their own client using the WebHDFS REST API.
> For #1, it is very difficult to manage and unnecessarily complicated for other applications
since Hadoop is not a lightweight library.  For #2, it is not easy to deal with security and
handle transient errors.
> Therefore, we propose adding a lightweight WebHDFS client as a separated library which
does not depend on Common and HDFS.  The client can be packaged as a standalone jar.  Other
applications simply add the jar to their classpath for using it.

This message was sent by Atlassian JIRA

View raw message