hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12756) Incorporate Aliyun OSS file system implementation
Date Fri, 15 Jul 2016 09:12:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379069#comment-15379069
] 

Steve Loughran commented on HADOOP-12756:
-----------------------------------------

-1 to grabbing user and password for this FS. your secrets end up being logged everywhere,
as lots of code: error messages, logs, etc all assume that the URIs are safe to print

We tried in HADOOP-3733 to strip out user:pass logging, but it doesn't work. Instead it does
best effort, tells the user off —and warns that it'll be removed from future versions.

If the current version of this patch does grab user:password, that'll need to be cut out.
We don't want to repeat the same security risk. Sorry

What is good is to use the {{Configuration.getPassword()}} to use the credential management
in Hadoop —the secrets will then be encrypted outside the file.

it could also be good to support the ability to specify passwords in the config as an optional
s3a.login.${endpoint}.${bucket}.user= and the same for password= values. That way you can
have the config files set up with different logins for different accounts, so do cross account
disctp work.

> Incorporate Aliyun OSS file system implementation
> -------------------------------------------------
>
>                 Key: HADOOP-12756
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12756
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 2.8.0, HADOOP-12756
>            Reporter: shimingfei
>            Assignee: shimingfei
>             Fix For: HADOOP-12756
>
>         Attachments: HADOOP-12756-v02.patch, HADOOP-12756.003.patch, HADOOP-12756.004.patch,
HADOOP-12756.005.patch, HADOOP-12756.006.patch, HCFS User manual.md, OSS integration.pdf,
OSS integration.pdf
>
>
> Aliyun OSS is widely used among China’s cloud users, but currently it is not easy to
access data laid on OSS storage from user’s Hadoop/Spark application, because of no original
support for OSS in Hadoop.
> This work aims to integrate Aliyun OSS with Hadoop. By simple configuration, Spark/Hadoop
applications can read/write data from OSS without any code change. Narrowing the gap between
user’s APP and data storage, like what have been done for S3 in Hadoop 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message