hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-3733) "s3:" URLs break when Secret Key contains a slash, even if encoded
Date Wed, 15 Jun 2016 13:45:09 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran updated HADOOP-3733:
-----------------------------------
    Attachment: HADOOP-3733-branch-2-004.patch

patch branch-2-004; fixes checkstyle.

Chris: thanks for the +1; I'm waiting for Ravi to do another attempt at trying to get this
to work. 

FWIW, I don' think people should be trying to use credentials on the CLI. This patch tries
to strip it from the URL and path, but they do creep out in error messages and stack traces.

> "s3:" URLs break when Secret Key contains a slash, even if encoded
> ------------------------------------------------------------------
>
>                 Key: HADOOP-3733
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3733
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 0.17.1, 2.0.2-alpha
>            Reporter: Stuart Sierra
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: HADOOP-3733-20130223T011025Z.patch, HADOOP-3733-branch-2-001.patch,
HADOOP-3733-branch-2-002.patch, HADOOP-3733-branch-2-003.patch, HADOOP-3733-branch-2-004.patch,
HADOOP-3733.patch, hadoop-3733.patch
>
>
> When using URLs of the form s3://ID:SECRET@BUCKET/ at the command line, distcp fails
if the SECRET contains a slash, even when the slash is URL-encoded as %2F.
> Say your AWS Access Key ID is RYWX12N9WCY42XVOL8WH
> And your AWS Secret Key is Xqj1/NMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
> And your bucket is called "mybucket"
> You can URL-encode the Secret KKey as Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv
> But this doesn't work:
> {noformat}
> $ bin/hadoop distcp file:///source  s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
> 08/07/09 15:05:22 INFO util.CopyFiles: srcPaths=[file:///source]
> 08/07/09 15:05:22 INFO util.CopyFiles: destPath=s3://RYWX12N9WCY42XVOL8WH:Xqj1%2FNMvKBhl1jqKlzbYJS66ua0e8z7Kkvptl9bv@mybucket/dest
> 08/07/09 15:05:23 WARN httpclient.RestS3Service: Unable to access bucket: mybucket
> org.jets3t.service.S3ServiceException: S3 HEAD request failed. ResponseCode=403, ResponseMessage=Forbidden
>         at org.jets3t.service.impl.rest.httpclient.RestS3Service.performRequest(RestS3Service.java:339)
> ...
> With failures, global counters are inaccurate; consider running with -i
> Copy failed: org.apache.hadoop.fs.s3.S3Exception: org.jets3t.service.S3ServiceException:
S3 PUT failed. XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The
request signature we calculated does not match the signature you provided. Check your key
and signing method.</Message>
>         at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.createBucket(Jets3tFileSystemStore.java:141)
> ...
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message