hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2178) Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
Date Thu, 20 Oct 2011 22:20:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13132111#comment-13132111
] 

Sanjay Radia commented on HDFS-2178:
------------------------------------

* 100 Continue issue
** I like arpits's suggestion since it does leak implementation limitations to the API; this
almost the same as a getCreateHandle and a getAppendHandle operations. That is simply submit
the "put" without the data and use the result as the handle. Alejandro, is this acceptable?
* I agree that the we want to keep the proxy and webhdfs APIs the same or almost the same
- but if an operation does not make sense for the proxy or if an operation does not make sense
for webhdfs we should allow such differences. For example, will the proxy ever redirect? Does
getDelegationToken make sense?
* APi - the webhdfs API has been derived from the original Hoop APi with changes made based
on feedback over the last month. I would like to start with the webhdfs api as currently in
truck and 205 as the stating point. Nicholas will post a document that describes the current
implementation.
* Code sharing - I agree - right now lets get the apis to match and then over time we can
move to a shared implementation.
* Pure proxy vs hdfs proxy. I agree that hdfs proxy has merit beyond pure proxy (e.g. proxy
for s3, authentication mapping etc). Would it make sense simply forward the webhdfs operations
directly as is to  webhdfs inside HDFS? This makes code sharing even easier. 
                
> Contributing Hoop to HDFS, replacement for HDFS proxy with read/write capabilities
> ----------------------------------------------------------------------------------
>
>                 Key: HDFS-2178
>                 URL: https://issues.apache.org/jira/browse/HDFS-2178
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 0.23.0
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>             Fix For: 0.23.0
>
>         Attachments: HDFS-2178.patch, HDFSoverHTTP-API.html, HdfsHttpAPI.pdf
>
>
> We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) for HDFS
Proxy.
> Hoop provides access to all Hadoop Distributed File System (HDFS) operations (read and
write) over HTTP/S.
> The Hoop server component is a REST HTTP gateway to HDFS supporting all file system operations.
It can be accessed using standard HTTP tools (i.e. curl and wget), HTTP libraries from different
programing languages (i.e. Perl, Java Script) as well as using the Hoop client. The Hoop server
component is a standard Java web-application and it has been implemented using Jersey (JAX-RS).
> The Hoop client component is an implementation of Hadoop FileSystem client that allows
using the familiar Hadoop filesystem API to access HDFS data through a Hoop server.
>   Repo: https://github.com/cloudera/hoop
>   Docs: http://cloudera.github.com/hoop
>   Blog: http://www.cloudera.com/blog/2011/07/hoop-hadoop-hdfs-over-http/
> Hoop is a Maven based project that depends on Hadoop HDFS and Alfredo (for Kerberos HTTP
SPNEGO authentication). 
> To make the integration easy, HDFS Mavenization (HDFS-2096) would have to be done first,
as well as the Alfredo contribution (HADOOP-7119).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message