hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen Montgomery (JIRA)" <j...@apache.org>
Subject [jira] [Reopened] (HDFS-8727) Allow using path style addressing for accessing the s3 endpoint
Date Tue, 22 Mar 2016 17:24:25 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Stephen Montgomery reopened HDFS-8727:
      Assignee:     (was: Andrew Baptist)

I'd like to re-open this ticket please. I've done some further digging into this and believe
that Andrew's original patch is still needed ie using a Hadoop S3A config property flag to
"switch on" path style access in the underlying Amazon S3 client. Overriding the custom S3A
endpoint has no effect (unless you specifically use an IPv4 address which is more a workaround/hack).

To force/trick the Amazon S3 client to use old path style access (instead of virtual hosting)
you can use dodgy bucket names (eg '..', '.-' in the name, caps etc) and IPv4 addresses for
the endpoint - see com.amazonaws.services.s3.AmazonS3Client. configRequest() method - pretty
much making sure that the DNS lookups will fail for syntactic reasons.

I'm happy to update Andrew's original patch and supply a test case, if needed. Like Andrew
mentioned, the test case will be of no real benefit as it will just exercising Amazon client
functionality. It's also hard to do as the AWS client is pretty inaccessible around confirming
the flag has been set.

Whats the process of re-opening this ticket? What Hadoop branch will this be targeted for
ie it looks that 2.8 one has all of the S3A fixes...?


> Allow using path style addressing for accessing the s3 endpoint
> ---------------------------------------------------------------
>                 Key: HDFS-8727
>                 URL: https://issues.apache.org/jira/browse/HDFS-8727
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.7.1
>            Reporter: Andrew Baptist
>              Labels: features
>         Attachments: hdfs-8728.patch.2
> There is no ability to specify using path style access for the s3 endpoint. There are
numerous non-amazon implementations of storage that support the amazon API's but only support
path style access such as Cleversafe and Ceph. Additionally in many environments it is difficult
to configure DNS correctly to get virtual host style addressing to work

This message was sent by Atlassian JIRA

View raw message