hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13336) S3A to support per-bucket configuration
Date Thu, 12 Jan 2017 11:31:03 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran updated HADOOP-13336:
          Resolution: Fixed
       Fix Version/s: 2.8.0
    Target Version/s:   (was: 2.9.0)
              Status: Resolved  (was: Patch Available)

This is now in Hadoop 2.8+, meaning everyone will get the ability to declare different endpoints,
auth and fadvice options for different buckets.

thank you to everyone who reviewed, from core design to the security auditing.

For reference, this was the merge strategy. I don't know if it was ideal, and it was slow/careful
going, but it worked (AFAIK)
# apply trunk branch
# cherry pick to branch-2; test/verify
# cherry pick to branch 2.8; test verify (and when problem found, fix and build new patch.
# cherry pick to branch 2.8.0
# fork off Hadoop-13345 branch, apply trunk 012 patch, _and generate diff between that and
the branch with the full HADOOP-13336-HADOOP-13445-013 patch_
# merge trunk into HADOOP-13345 branch
# apply the diff previously generated
# commit and push

Yes, this did take >1 attempt

> S3A to support per-bucket configuration
> ---------------------------------------
>                 Key: HADOOP-13336
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13336
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>             Fix For: 2.8.0
>         Attachments: HADOOP-13336-006.patch, HADOOP-13336-007.patch, HADOOP-13336-010.patch,
HADOOP-13336-011.patch, HADOOP-13336-HADOOP-13345-001.patch, HADOOP-13336-HADOOP-13345-002.patch,
HADOOP-13336-HADOOP-13345-003.patch, HADOOP-13336-HADOOP-13345-004.patch, HADOOP-13336-HADOOP-13345-005.patch,
HADOOP-13336-HADOOP-13345-006.patch, HADOOP-13336-HADOOP-13345-008.patch, HADOOP-13336-HADOOP-13345-009.patch,
> S3a now supports different regions, by way of declaring the endpoint —but you can't
do things like read in one region, write back in another (e.g. a distcp backup), because only
one region can be specified in a configuration.
> If s3a supported region declaration in the URL, e.g. s3a://b1.frankfurt s3a://b2.seol
, then this would be possible. 
> Swift does this with a full filesystem binding/config: endpoints, username, etc, in the
XML file. Would we need to do that much? It'd be simpler initially to use a domain suffix
of a URL to set the region of a bucket from the domain and have the aws library sort the details
out itself, maybe with some config options for working with non-AWS infra

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message