hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shimingfei (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
Date Thu, 04 Feb 2016 03:04:40 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131629#comment-15131629

shimingfei commented on HADOOP-12756:

Thanks Chris. it is very helpful.

1. The intention of this work was to make Spark/Hadoop applications be able to read/write
data from OSS, not completely run Hadoop/Spark over it, because of some limitation on OSS(or
object stores). the FileSystem API is offered, just like S3
2. Clients should hold credentials, proxy is just used to access the OSS service as an configuration
of client.
3. Thanks for your suggestions, we will follow that specification.
4. yes, OSS support the mapping, we will add more description for this.
5. sure, we will offer more docs for end users, and currently the approach of renaming in
OSS is copy and delete, like S3.
6. currently, our implementation doesn't have emulation capability, We will look into it.

> Incorporate Aliyun OSS file system implementation
> -------------------------------------------------
>                 Key: HADOOP-12756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12756
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: shimingfei
>            Assignee: shimingfei
>         Attachments: OSS integration.pdf
> Aliyun OSS is widely used among China’s cloud users, but currently it is not easy to
access data laid on OSS storage from user’s Hadoop/Spark application, because of no original
support for OSS in Hadoop.
> This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, Spark/Hadoop
applications can read/write data from OSS without any code change. Narrowing the gap between
user’s APP and data storage, like what have been done for S3 in Hadoop 

This message was sent by Atlassian JIRA

View raw message