hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13287) TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains '+'.
Date Fri, 17 Jun 2016 21:49:05 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Nauroth updated HADOOP-13287:
    Attachment: HADOOP-13287.001.patch

My test run was on branch-2.8 against an S3 bucket in US-west-2.  What I saw happening was
a double decoding in {{S3xLoginHelper#extractLoginDetails}}:

  public static Login extractLoginDetails(URI name) {
    try {
      String authority = name.getAuthority();
        String password = URLDecoder.decode(login.substring(loginSplit + 1),

According to the JavaDocs for [{{URI#getAuthority}}|http://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getAuthority()],
it performs decoding already on the output.  Then we do a second explicit decoding by calling
{{URLDecoder#decode}}.  First, {{getAuthority}} translates "%2B" to "\+".  Then, {{URLDecoder#decode}}
translates "\+" to " ", which isn't correct for the credentials.

However, this appear to be only a problem in the JUnit test runs.  I also built a distro and
tested manually with URIs that contain '+' encoded as "%2B", and that worked just fine.  The
reason it works fine there is because of different encoding rules applied by round-tripping
through a {{Path}} before the {{FileSystem#get}} call gets triggered.  With {{Path}}, the
'+' gets double-encoded to "%252B", so double-decoding at the S3A layer is correct logic.

To make this work, the test should follow the same encoding as would be used on the CLI. 
The attached path switches from constructing a {{URI}} to constructing a {{Path}}.  I switched
the exception stifling logic to catch {{IllegalArgumentException}}, becauses that's what {{Path}}
throws.  With this, the test passes with a secret containing a '+'.  [~stevel@apache.org]
or [~raviprak], I understand one of you might have a secret with a '/' from your work on HADOOP-3733.
 Would you mind testing this patch to make sure the test still passes with '/'?

> TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains '+'.
> -------------------------------------------------------------------------------
>                 Key: HADOOP-13287
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13287
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3, test
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>            Priority: Minor
>         Attachments: HADOOP-13287.001.patch
> HADOOP-3733 fixed accessing S3A with credentials on the command line for an AWS secret
key containing a '/'.  The patch added a new test suite: {{TestS3ACredentialsInURL}}.  One
of the tests fails if your AWS secret key contains a '+'.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message