hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14391) s3a: auto-detect region for bucket and use right endpoint
Date Thu, 01 Jun 2017 10:36:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032764#comment-16032764
] 

Steve Loughran commented on HADOOP-14391:
-----------------------------------------

One problem here is that the rate of new AWS regions coming online invariably lags rate of
update of SDK versions. We're lucky that we don't use the enumerated values, otherwise there'd
be pressure to upgrade whenever new locations like Mumbai and London go live. If any auto
discovery depends on the SDK knowing the regions and their endpoints, it is susceptible to
this probem

> s3a: auto-detect region for bucket and use right endpoint
> ---------------------------------------------------------
>
>                 Key: HADOOP-14391
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14391
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: s3
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
>
> Specifying the S3A endpoint ({{fs.s3a.endpoint}}) is
> - *required* for regions which only support v4 authentication
> - A good practice for all regions.
> The user experience of having to configure endpoints is not great.  Often it is neglected
and leads to additional cost, reduced performance, or failures for v4 auth regions.
> I want to explore an option which, when enabled, auto-detects the region for an s3 bucket
and uses the proper endpoint.  Not sure if this is possible or anyone has looked into it yet.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message