hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
Date Mon, 31 Jul 2017 15:26:01 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107425#comment-16107425
] 

Vinayakumar B commented on HADOOP-3733:
---------------------------------------

Stripping down the userinfo part of provided uri in the filesystem's {{uri}} makes the Distcp
fail in following case.
1. AccessKey and Secret Key are provided in the URI itself (ideally it should not, but for
ease of use, old application uses old style uris)
2. While doing distcp of directory, {{fs.listStatus()}} results in paths WITHOUT these AccessKeys
and Secrets ( because fs.makeQualified() uses {{uri}} without AK:SK)
3. Which eventually fails in Maps as there is no other way of credentials are given.

As Release note says, supporting userinfo on URI itself would be useful for distcp, but currently
its breaking the existing functionality.

[~stevel@apache.org]/[~cnauroth], whether old functionality should be added back?

> "s3:" URLs break when Secret Key contains a slash, even if encoded
> ------------------------------------------------------------------
>
>                 Key: HADOOP-3733
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3733
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.17.1, 2.0.2-alpha
>            Reporter: Stuart Sierra
>            Assignee: Steve Loughran
>            Priority: Minor
>             Fix For: 2.8.0, 3.0.0-alpha1
>
>         Attachments: HADOOP-3733-20130223T011025Z.patch, HADOOP-3733-branch-2-001.patch,
HADOOP-3733-branch-2-002.patch, HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch,
HADOOP-3733-branch-2-005.patch, HADOOP-3733-branch-2-006.patch, HADOOP-3733-branch-2-007.patch,
hadoop-3733.patch, HADOOP-3733.patch
>
>
> When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, distcp fails
if the SECRET contains a slash, even when the slash is URL-encoded as %2F.
> Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
> And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
> And your bucket is called "mybucket"
> You can URL-encode the Secret KKey as Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
> But this doesn't work:
> {noformat}
> $ bin/hadoop distcp file:///source  s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
> 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
> 08/07/09 15:05:22 INFO util.CopyFiles: destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
> 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: mybucket
> org.jets3t.service.S3ServiceException: S3 HEAD request failed. ResponseCode=403, ResponseMessage=Forbidden
>         at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
> ...
> With failures, global counters are inaccurate; consider running with -i
> Copy failed: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException:
S3 PUT failed. XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The
request signature we calculated does not match the signature you provided. Check your key
and signing method.</Message>
>         at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
> ...
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message