hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
Date Fri, 27 May 2016 05:55:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15303561#comment-15303561
] 

Chris Nauroth commented on HADOOP-12756:
----------------------------------------

The situation is similar to the testing challenges we face for other similar modules: hadoop-aws,
hadoop-azure and hadoop-openstack.  Right now, these modules don't really get tested during
pre-commit, except for a few true unit tests that don't require integration with an external
service.  Instead, it's up to reviewers to get credentials to the external service, either
using a personal account or an employer's account, and configure the tests to run from their
development environments.

There is enough activity happening in these modules that it would be in our best interest
to get the tests running from pre-commit.  I haven't put much thought into how this would
work.  Kai is right that the credentials need to go somewhere accessible by each Jenkins host
that runs a Hadoop pre-commit build.  However, we wouldn't want those credentials accessible
by the whole Internet or really even the whole Apache contributor community.  I expect it
will take coordination with the Apache infra team to get this done correctly.

> Incorporate Aliyun OSS file system implementation
> -------------------------------------------------
>
>                 Key: HADOOP-12756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12756
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 2.8.0
>            Reporter: shimingfei
>            Assignee: shimingfei
>         Attachments: HADOOP-12756-v02.patch, HCFS User manual.md, OSS integration.pdf,
OSS integration.pdf
>
>
> Aliyun OSS is widely used among China’s cloud users, but currently it is not easy to
access data laid on OSS storage from user’s Hadoop/Spark application, because of no original
support for OSS in Hadoop.
> This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, Spark/Hadoop
applications can read/write data from OSS without any code change. Narrowing the gap between
user’s APP and data storage, like what have been done for S3 in Hadoop 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message