hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
Date Mon, 25 Apr 2016 09:39:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15256146#comment-15256146
] 

Steve Loughran commented on HADOOP-12756:
-----------------------------------------

# I agree: name change, especially as OSS is also the acronym "Open Source Software". Make
it hadoo-aliyun (or some other obvious name). Keeping it bigger than just OSS allows for more
features for the platform to go in later
# There's an open JIRA on incrementing http components; HADOOP-12767; I'm expecting this to
go in for Hadoop 2.9. which is what this patch can target (hence: work directly with hadoop-trunk
for your dev & patches, not branch-2.8)
# All version dependencies must be declared in {{hadoop-project/pom.xml}}; it's how we make
sure versions are consistent.
# regarding passing down usernames, this must be done via the {{test/resources/auth-keys.xml}}
file. Look at the aws or openstack modules to see how the tests are automatically skipped
if undefined. See also [how to keep your credentials private|http://steveloughran.blogspot.co.uk/2016/04/testing-against-s3-and-object-stores.html].
Using the hadoop XML files lets you also test credential provider integration, which we'll
also expect
# Have a look at the s3a work, especially those items in phase i: stabilisation, HADOOP-11571,
Make sure that the patch avoids those same problems (e.g how to close vs abort streams, swallowing
FileNotFoundExceptions during the final delete phase). S3a phase II, HADOOP-11694 contains
some other bugs, but is otherwise performance work. It's probably best to wait one iteration
before doing the performance version, get things stable first.
# nice to see all the tests!










> Incorporate Aliyun OSS file system implementation
> -------------------------------------------------
>
>                 Key: HADOOP-12756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12756
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>            Reporter: shimingfei
>            Assignee: shimingfei
>         Attachments: 0001-OSS-filesystem-integration-with-Hadoop.patch, HCFS User manual.md,
OSS integration.pdf, OSS integration.pdf
>
>
> Aliyun OSS is widely used among China’s cloud users, but currently it is not easy to
access data laid on OSS storage from user’s Hadoop/Spark application, because of no original
support for OSS in Hadoop.
> This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, Spark/Hadoop
applications can read/write data from OSS without any code change. Narrowing the gap between
user’s APP and data storage, like what have been done for S3 in Hadoop 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message