hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
Date Thu, 23 Jun 2016 13:21:16 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346422#comment-15346422
] 

Steve Loughran commented on HADOOP-12756:
-----------------------------------------

I'm really pleased with the progress here —and have to apologise for not putting in the
time it deserves reviewing it: [~drankye] has been handling that well.

How should we progress here? I think we should consider some intermediate milestone for it
to be ready for a feature branch, then iterate on that for broader testing and review. It's
never going to be "perfect" when it goes it —there's always scope for improvement— i
think we initially should be happy all builds and runs, the contract tests can pass, that
initial downstream runs don't find problems and then do what we've done with the other object
stores: ship it and see what surfaces in the field. At a guess, it'll be authentication.

codewise, 

- do make sure that {{FileSystem.close()}} always works, even before {{initialize()}} is invoked.
I'm not sure that holds right now.
- I think the password pickup from the config should support CredentialProviders, the way
that that S3A does. That provides a significantly more secure option. Having it in there from
the outset lets the docs recommend this from the outset.

> Incorporate Aliyun OSS file system implementation
> -------------------------------------------------
>
>                 Key: HADOOP-12756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12756
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 2.8.0
>            Reporter: shimingfei
>            Assignee: shimingfei
>         Attachments: HADOOP-12756-v02.patch, HADOOP-12756.003.patch, HADOOP-12756.004.patch,
HADOOP-12756.005.patch, HCFS User manual.md, OSS integration.pdf, OSS integration.pdf
>
>
> Aliyun OSS is widely used among China’s cloud users, but currently it is not easy to
access data laid on OSS storage from user’s Hadoop/Spark application, because of no original
support for OSS in Hadoop.
> This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, Spark/Hadoop
applications can read/write data from OSS without any code change. Narrowing the gap between
user’s APP and data storage, like what have been done for S3 in Hadoop 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message