From common-issues-return-147429-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Wed Jan 31 22:07:07 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 8BF7618066D for ; Wed, 31 Jan 2018 22:07:07 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 7BD32160C2C; Wed, 31 Jan 2018 21:07:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C1703160C42 for ; Wed, 31 Jan 2018 22:07:06 +0100 (CET) Received: (qmail 12558 invoked by uid 500); 31 Jan 2018 21:07:05 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 12547 invoked by uid 99); 31 Jan 2018 21:07:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Jan 2018 21:07:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 45382C0494 for ; Wed, 31 Jan 2018 21:07:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.811 X-Spam-Level: X-Spam-Status: No, score=-109.811 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 7C59w5y1KsVX for ; Wed, 31 Jan 2018 21:07:04 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id D64A55F3E1 for ; Wed, 31 Jan 2018 21:07:03 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 27F08E0220 for ; Wed, 31 Jan 2018 21:07:03 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 8238E24106 for ; Wed, 31 Jan 2018 21:07:02 +0000 (UTC) Date: Wed, 31 Jan 2018 21:07:02 +0000 (UTC) From: "Steve Loughran (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HADOOP-14620) S3A authentication failure for regions other than us-east-1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-14620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14620: ------------------------------------ Priority: Minor (was: Major) > S3A authentication failure for regions other than us-east-1 > ----------------------------------------------------------- > > Key: HADOOP-14620 > URL: https://issues.apache.org/jira/browse/HADOOP-14620 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 > Affects Versions: 2.8.0, 2.7.3 > Reporter: Ilya Fourmanov > Priority: Minor > Attachments: s3-403.txt > > > hadoop fs s3a:// operations fail authentication for s3 buckets hosted in regions other than default us-east-1 > Steps to reproduce: > # create s3 bucket in eu-west-1 > # Using IAM instance profile or fs.s3a.access.key/fs.s3a.secret.key run following command: > {code} > hadoop --loglevel DEBUG -D fs.s3a.endpoint=s3.eu-west-1.amazonaws.com -ls s3a://your-eu-west-1-hosted-bucket/ > {code} > Expected behaviour: > You will see listing of the bucket > Actual behaviour: > You will get 403 Authentication Denied response for AWS S3. > Reason is mismatch in string to sign as defined in http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html provided by hadoop and expected by AWS. > If you use https://aws.amazon.com/code/199 to analyse StringToSignBytes returned by AWS, you will see that AWS expects CanonicalizedResource to be in form /your-eu-west-1-hosted-bucket{color:red}.s3.eu-west-1.amazonaws.com{color}/. > Hadoop provides it as /your-eu-west-1-hosted-bucket/ > Note that AWS documentation doesn't explicitly state that endpoint or full dns address should be appended to CanonicalizedResource however practice shows it is actually required. > I've also submitted this to AWS for them to correct behaviour or documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-issues-help@hadoop.apache.org