hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
Date Mon, 25 Apr 2016 09:39:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15256146#comment-15256146

Steve Loughran commented on HADOOP-12756:

# I agree: name change, especially as OSS is also the acronym "Open Source Software". Make
it hadoo-aliyun (or some other obvious name). Keeping it bigger than just OSS allows for more
features for the platform to go in later
# There's an open JIRA on incrementing http components; HADOOP-12767; I'm expecting this to
go in for Hadoop 2.9. which is what this patch can target (hence: work directly with hadoop-trunk
for your dev & patches, not branch-2.8)
# All version dependencies must be declared in {{hadoop-project/pom.xml}}; it's how we make
sure versions are consistent.
# regarding passing down usernames, this must be done via the {{test/resources/auth-keys.xml}}
file. Look at the aws or openstack modules to see how the tests are automatically skipped
if undefined. See also [how to keep your credentials private|http://steveloughran.blogspot.co.uk/2016/04/testing-against-s3-and-object-stores.html].
Using the hadoop XML files lets you also test credential provider integration, which we'll
also expect
# Have a look at the s3a work, especially those items in phase i: stabilisation, HADOOP-11571,
Make sure that the patch avoids those same problems (e.g how to close vs abort streams, swallowing
FileNotFoundExceptions during the final delete phase). S3a phase II, HADOOP-11694 contains
some other bugs, but is otherwise performance work. It's probably best to wait one iteration
before doing the performance version, get things stable first.
# nice to see all the tests!

> Incorporate Aliyun OSS file system implementation
> -------------------------------------------------
>                 Key: HADOOP-12756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12756
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: shimingfei
>            Assignee: shimingfei
>         Attachments: 0001-OSS-filesystem-integration-with-Hadoop.patch, HCFS User manual.md,
OSS integration.pdf, OSS integration.pdf
> Aliyun OSS is widely used among China’s cloud users, but currently it is not easy to
access data laid on OSS storage from user’s Hadoop/Spark application, because of no original
support for OSS in Hadoop.
> This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, Spark/Hadoop
applications can read/write data from OSS without any code change. Narrowing the gap between
user’s APP and data storage, like what have been done for S3 in Hadoop 

This message was sent by Atlassian JIRA

View raw message