From common-issues-return-154289-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Fri Jun 29 13:36:09 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id DECB4180663 for ; Fri, 29 Jun 2018 13:36:07 +0200 (CEST) Received: (qmail 40732 invoked by uid 500); 29 Jun 2018 11:36:06 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 40721 invoked by uid 99); 29 Jun 2018 11:36:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jun 2018 11:36:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 79D8DC01C4 for ; Fri, 29 Jun 2018 11:36:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id L2jcdabtJS31 for ; Fri, 29 Jun 2018 11:36:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 9E8895F3F0 for ; Fri, 29 Jun 2018 11:36:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id BD452E0A31 for ; Fri, 29 Jun 2018 11:36:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 3278D21840 for ; Fri, 29 Jun 2018 11:36:00 +0000 (UTC) Date: Fri, 29 Jun 2018 11:36:00 +0000 (UTC) From: "Steve Loughran (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-15573) s3guard set-capacity to not retry on an access denied exception MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-15573?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16= 527493#comment-16527493 ]=20 Steve Loughran commented on HADOOP-15573: ----------------------------------------- {code} bin/hadoop s3guard $ARN set-capacity -read 10 $IRL 2018-06-29 12:31:45,179 [main] DEBUG s3guard.S3GuardTool (S3GuardTool.java:= run(1458)) - Executing command set-capacity Read capacity set to 10 2018-06-29 12:31:45,207 [main] DEBUG s3guard.S3GuardTool (S3GuardTool.java:= initS3AFileSystem(301)) - updated bucket store option org.apache.hadoop.fs.= s3a.s3guard.NullMetadataStore 2018-06-29 12:31:45,974 [main] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:= initialize(237)) - Initializing S3AFileSystem for hwdev-steve-ireland-new 2018-06-29 12:31:45,976 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateB= ucketOptions(1001)) - Propagating entries under fs.s3a.bucket.hwdev-steve-i= reland-new. 2018-06-29 12:31:46,055 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateB= ucketOptions(1022)) - Updating fs.s3a.endpoint from [core-site.xml] 2018-06-29 12:31:46,055 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateB= ucketOptions(1022)) - Updating fs.s3a.committer.magic.enabled from [core-si= te.xml] 2018-06-29 12:31:46,056 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateB= ucketOptions(1022)) - Updating fs.s3a.metadatastore.impl from [S3AUtils] 2018-06-29 12:31:46,260 [main] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSC= redentialProvider(628)) - Credential provider class is org.apache.hadoop.fs= .s3a.auth.AssumedRoleCredentialProvider 2018-06-29 12:31:46,261 [main] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSC= redentialProvider(628)) - Credential provider class is org.apache.hadoop.fs= .s3a.SimpleAWSCredentialsProvider 2018-06-29 12:31:46,269 [main] DEBUG auth.AssumedRoleCredentialProvider (As= sumedRoleCredentialProvider.java:(117)) - AssumedRoleCredentialProvid= er{role=3D'arn:aws:iam::980678866538:role/stevel-s3-restricted', session'st= evel', duration=3D1800} 2018-06-29 12:31:46,271 [main] DEBUG auth.AssumedRoleCredentialProvider (As= sumedRoleCredentialProvider.java:(130)) - Credentials to obtain role = credentials: AWSCredentialProviderList: SimpleAWSCredentialsProvider 2018-06-29 12:31:46,827 [main] DEBUG s3a.AWSCredentialProviderList (AWSCred= entialProviderList.java:getCredentials(122)) - Using credentials from Simpl= eAWSCredentialsProvider 2018-06-29 12:31:47,566 [main] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSC= redentialProviderSet(572)) - For URI s3a://hwdev-steve-ireland-new//, using= credentials AWSCredentialProviderList: AssumedRoleCredentialProvider{role= =3D'arn:aws:iam::980678866538:role/stevel-s3-restricted', session'stevel', = duration=3D1800} 2018-06-29 12:31:47,567 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.connection.maximum is 15 2018-06-29 12:31:47,567 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.attempts.maximum is 20 2018-06-29 12:31:47,567 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.connection.establish.timeout is 5000 2018-06-29 12:31:47,568 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.connection.timeout is 5000 2018-06-29 12:31:47,568 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.socket.send.buffer is 65536 2018-06-29 12:31:47,568 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.socket.recv.buffer is 32678 2018-06-29 12:31:47,570 [main] DEBUG s3a.S3AFileSystem (DefaultS3ClientFact= ory.java:initUserAgent(183)) - Using User-Agent: Hadoop 3.2.0-SNAPSHOT 2018-06-29 12:31:47,637 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.paging.maximum is 5000 2018-06-29 12:31:47,640 [main] DEBUG s3a.S3AUtils (S3AUtils.java:longBytesO= ption(887)) - Value of fs.s3a.block.size is 33554432 2018-06-29 12:31:47,640 [main] DEBUG s3a.S3AUtils (S3AUtils.java:longBytesO= ption(887)) - Value of fs.s3a.readahead.range is 524288 2018-06-29 12:31:47,640 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.max.total.tasks is 5 2018-06-29 12:31:47,641 [main] DEBUG s3a.S3AUtils (S3AUtils.java:longOption= (865)) - Value of fs.s3a.threads.keepalivetime is 60 2018-06-29 12:31:47,656 [main] DEBUG s3a.AWSCredentialProviderList (AWSCred= entialProviderList.java:getCredentials(122)) - Using credentials from Assum= edRoleCredentialProvider{role=3D'arn:aws:iam::980678866538:role/stevel-s3-r= estricted', session'stevel', duration=3D1800} 2018-06-29 12:31:47,836 [main] DEBUG s3a.S3AUtils (S3AUtils.java:getEncrypt= ionAlgorithm(1253)) - Data is unencrypted 2018-06-29 12:31:47,837 [main] DEBUG s3a.S3AUtils (S3AUtils.java:getEncrypt= ionAlgorithm(1256)) - Using SSE-C with empty key 2018-06-29 12:31:47,837 [main] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:= initialize(313)) - Input fadvise policy =3D normal 2018-06-29 12:31:47,837 [main] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:= initialize(317)) - Filesystem support for magic committers is enabled 2018-06-29 12:31:47,843 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.fast.upload.active.blocks is 4 2018-06-29 12:31:47,843 [main] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:= initialize(333)) - Using S3ABlockOutputStream with buffer =3D disk; block= =3D8388608; queue limit=3D4 2018-06-29 12:31:47,846 [main] DEBUG s3guard.S3Guard (S3Guard.java:getMetad= ataStoreClass(125)) - Metastore option source fs.s3a.bucket.hwdev-steve-ire= land-new.metadatastore.impl via [S3AUtils] 2018-06-29 12:31:47,847 [main] DEBUG s3guard.S3Guard (S3Guard.java:getMetad= ataStore(97)) - Using NullMetadataStore metadata store for s3a filesystem 2018-06-29 12:31:47,847 [main] DEBUG s3a.S3AUtils (S3AUtils.java:longOption= (865)) - Value of fs.s3a.multipart.purge.age is 3600000 2018-06-29 12:31:47,924 [main] DEBUG s3guard.DynamoDBMetadataStore (DynamoD= BMetadataStore.java:initialize(277)) - Inferring DynamoDB region from S3 bu= cket: eu-west-1 2018-06-29 12:31:47,924 [main] DEBUG s3guard.DynamoDBMetadataStore (DynamoD= BMetadataStore.java:createDynamoDB(255)) - Creating DynamoDB client class o= rg.apache.hadoop.fs.s3a.s3guard.DynamoDBClientFactory$DefaultDynamoDBClient= Factory with S3 region eu-west-1 2018-06-29 12:31:47,924 [main] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSC= redentialProvider(628)) - Credential provider class is org.apache.hadoop.fs= .s3a.auth.AssumedRoleCredentialProvider 2018-06-29 12:31:47,924 [main] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSC= redentialProvider(628)) - Credential provider class is org.apache.hadoop.fs= .s3a.SimpleAWSCredentialsProvider 2018-06-29 12:31:47,925 [main] DEBUG auth.AssumedRoleCredentialProvider (As= sumedRoleCredentialProvider.java:(117)) - AssumedRoleCredentialProvid= er{role=3D'arn:aws:iam::980678866538:role/stevel-s3-restricted', session'st= evel', duration=3D1800} 2018-06-29 12:31:47,925 [main] DEBUG auth.AssumedRoleCredentialProvider (As= sumedRoleCredentialProvider.java:(130)) - Credentials to obtain role = credentials: AWSCredentialProviderList: SimpleAWSCredentialsProvider 2018-06-29 12:31:47,927 [main] DEBUG s3a.AWSCredentialProviderList (AWSCred= entialProviderList.java:getCredentials(122)) - Using credentials from Simpl= eAWSCredentialsProvider 2018-06-29 12:31:48,378 [main] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSC= redentialProviderSet(572)) - For URI (null URI), using credentials AWSCrede= ntialProviderList: AssumedRoleCredentialProvider{role=3D'arn:aws:iam::98067= 8866538:role/stevel-s3-restricted', session'stevel', duration=3D1800} 2018-06-29 12:31:48,379 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.connection.maximum is 15 2018-06-29 12:31:48,379 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.attempts.maximum is 20 2018-06-29 12:31:48,379 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.connection.establish.timeout is 5000 2018-06-29 12:31:48,379 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.connection.timeout is 5000 2018-06-29 12:31:48,380 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.socket.send.buffer is 65536 2018-06-29 12:31:48,380 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(= 844)) - Value of fs.s3a.socket.recv.buffer is 32678 2018-06-29 12:31:48,380 [main] DEBUG s3a.S3AFileSystem (DefaultS3ClientFact= ory.java:initUserAgent(183)) - Using User-Agent: Hadoop 3.2.0-SNAPSHOT 2018-06-29 12:31:48,382 [main] DEBUG s3guard.DynamoDBClientFactory (DynamoD= BClientFactory.java:createDynamoDBClient(84)) - Creating DynamoDB client in= region eu-west-1 2018-06-29 12:31:48,472 [main] DEBUG s3guard.DynamoDBMetadataStore (DynamoD= BMetadataStore.java:initTable(893)) - Binding to table hwdev-steve-ireland-= new 2018-06-29 12:31:48,519 [main] DEBUG s3a.AWSCredentialProviderList (AWSCred= entialProviderList.java:getCredentials(122)) - Using credentials from Assum= edRoleCredentialProvider{role=3D'arn:aws:iam::980678866538:role/stevel-s3-r= estricted', session'stevel', duration=3D1800} 2018-06-29 12:31:48,867 [main] DEBUG s3guard.DynamoDBMetadataStore (DynamoD= BMetadataStore.java:initTable(895)) - Table state: {AttributeDefinitions: [= {AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeTy= pe: S}],TableName: hwdev-steve-ireland-new,KeySchema: [{AttributeName: pare= nt,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTI= VE,CreationDateTime: Fri Jun 29 12:20:03 BST 2018,ProvisionedThroughput: {N= umberOfDecreasesToday: 0,ReadCapacityUnits: 500,WriteCapacityUnits: 100},Ta= bleSizeBytes: 0,ItemCount: 0,TableArn: arn:aws:dynamodb:eu-west-1:980678866= 538:table/hwdev-steve-ireland-new,TableId: 99dfc220-a142-4816-9b3b-bfed91de= 2d30,} 2018-06-29 12:31:48,917 [main] DEBUG s3guard.DynamoDBMetadataStore (DynamoD= BMetadataStore.java:initTable(920)) - Using existing DynamoDB table hwdev-s= teve-ireland-new in region eu-west-1 created Fri Jun 29 12:20:13 BST 2018 2018-06-29 12:31:48,918 [main] INFO s3guard.S3GuardTool (S3GuardTool.java:= initMetadataStore(270)) - Metadata store DynamoDBMetadataStore{region=3Deu-= west-1, tableName=3Dhwdev-steve-ireland-new} is initialized. 2018-06-29 12:31:48,954 [main] INFO s3guard.DynamoDBMetadataStore (DynamoD= BMetadataStore.java:updateParameters(1200)) - Current table capacity is rea= d: 500, write: 100 2018-06-29 12:31:48,954 [main] INFO s3guard.DynamoDBMetadataStore (DynamoD= BMetadataStore.java:updateParameters(1202)) - Changing capacity of table to= read: 10, write: 100 2018-06-29 12:31:49,041 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 1 2018-06-29 12:31:49,546 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #1 2018-06-29 12:31:49,588 [main] INFO s3guard.DynamoDBMetadataStore (DynamoD= BMetadataStore.java:retryEvent(1251)) - Retrying ProvisionTable on hwdev-st= eve-ireland-new: org.apache.hadoop.fs.s3a.AWSServiceIOException: ProvisionT= able on hwdev-steve-ireland-new: com.amazonaws.services.dynamodbv2.model.Am= azonDynamoDBException: User: arn:aws:sts::980678866538:assumed-role/stevel-= s3-restricted/stevel is not authorized to perform: dynamodb:UpdateTable on = resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland= -new (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDenied= Exception; Request ID: KHFS1RER89LU8OAC6967B5U8S7VV4KQNSO5AEMVJF66Q9ASUAAJG= ): User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel= is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dy= namodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new (Service: Amazo= nDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request I= D: KHFS1RER89LU8OAC6967B5U8S7VV4KQNSO5AEMVJF66Q9ASUAAJG) 2018-06-29 12:31:49,589 [main] DEBUG s3guard.DynamoDBMetadataStore (DynamoD= BMetadataStore.java:retryEvent(1252)) - Retrying ProvisionTable on hwdev-st= eve-ireland-new org.apache.hadoop.fs.s3a.AWSServiceIOException: ProvisionTable on hwdev-ste= ve-ireland-new: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBExcep= tion: User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/ste= vel is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws= :dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new (Service: Am= azonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Reques= t ID: KHFS1RER89LU8OAC6967B5U8S7VV4KQNSO5AEMVJF66Q9ASUAAJG): User: arn:aws:= sts::980678866538:assumed-role/stevel-s3-restricted/stevel is not authorize= d to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:eu-west-1:= 980678866538:table/hwdev-steve-ireland-new (Service: AmazonDynamoDBv2; Stat= us Code: 400; Error Code: AccessDeniedException; Request ID: KHFS1RER89LU8O= AC6967B5U8S7VV4KQNSO5AEMVJF66Q9ASUAAJG) =09at org.apache.hadoop.fs.s3a.S3AUtils.translateDynamoDBException(S3AUtils= .java:389) =09at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:18= 1) =09at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111) =09at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260) =09at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317) =09at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256) =09at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:188) =09at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:210) =09at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.provisionTable= (DynamoDBMetadataStore.java:1080) =09at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.provisionTable= Blocking(DynamoDBMetadataStore.java:1095) =09at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.updateParamete= rs(DynamoDBMetadataStore.java:1204) =09at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$SetCapacity.run(S3GuardT= ool.java:498) =09at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:350= ) =09at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) =09at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:148= 9) =09at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:14= 98) Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:= User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel i= s not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dyna= modb:eu-west-1:980678866538:table/hwdev-steve-ireland-new (Service: AmazonD= ynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID:= KHFS1RER89LU8OAC6967B5U8S7VV4KQNSO5AEMVJF66Q9ASUAAJG) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorRespon= se(AmazonHttpClient.java:1639) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest= (AmazonHttpClient.java:1304) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(Ama= zonHttpClient.java:1056) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonH= ttpClient.java:743) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(= AmazonHttpClient.java:717) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHtt= pClient.java:699) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(Amazon= HttpClient.java:667) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execu= te(AmazonHttpClient.java:649) =09at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513= ) =09at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(Amazo= nDynamoDBClient.java:2925) =09at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonD= ynamoDBClient.java:2901) =09at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeUpdateT= able(AmazonDynamoDBClient.java:2757) =09at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateTable(Am= azonDynamoDBClient.java:2733) =09at com.amazonaws.services.dynamodbv2.document.Table.updateTable(Table.ja= va:371) =09at com.amazonaws.services.dynamodbv2.document.Table.updateTable(Table.ja= va:463) =09at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.lambda$provisi= onTable$7(DynamoDBMetadataStore.java:1083) =09at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$2(Invoker.java:190) =09at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109) =09... 13 more 2018-06-29 12:31:49,591 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 2 2018-06-29 12:31:50,093 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #2 2018-06-29 12:31:50,136 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 3 2018-06-29 12:31:50,638 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #3 2018-06-29 12:31:50,684 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 4 2018-06-29 12:31:51,184 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #4 2018-06-29 12:31:51,228 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 5 2018-06-29 12:31:51,730 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #5 2018-06-29 12:31:51,774 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 6 2018-06-29 12:31:52,276 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #6 2018-06-29 12:31:52,319 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 7 2018-06-29 12:31:52,822 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #7 2018-06-29 12:31:52,865 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 8 2018-06-29 12:31:53,367 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #8 2018-06-29 12:31:53,411 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 9 2018-06-29 12:31:53,915 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #9 2018-06-29 12:31:53,959 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 10 2018-06-29 12:31:54,461 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #10 2018-06-29 12:31:54,505 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 11 2018-06-29 12:31:55,009 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #11 2018-06-29 12:31:55,052 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 12 2018-06-29 12:31:55,556 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #12 2018-06-29 12:31:55,605 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 13 2018-06-29 12:31:56,109 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #13 2018-06-29 12:31:56,153 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 14 2018-06-29 12:31:56,658 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #14 2018-06-29 12:31:56,701 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 15 2018-06-29 12:31:57,202 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #15 2018-06-29 12:31:57,245 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 16 2018-06-29 12:31:57,750 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #16 2018-06-29 12:31:57,792 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 17 2018-06-29 12:31:58,298 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #17 2018-06-29 12:31:58,342 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 18 2018-06-29 12:31:58,847 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #18 2018-06-29 12:31:58,889 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 19 2018-06-29 12:31:59,394 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #19 2018-06-29 12:31:59,438 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageSt= atistics.java:incrementCounter(63)) - ignored_errors +=3D 1 -> 20 2018-06-29 12:31:59,943 [main] DEBUG s3a.Invoker (Invoker.java:retryUntrans= lated(314)) - retry #20 org.apache.hadoop.fs.s3a.AWSServiceIOException: ProvisionTable on hwdev-ste= ve-ireland-new: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBExcep= tion: User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/ste= vel is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws= :dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new (Service: Am= azonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Reques= t ID: 2I6CMPEON2V56JKK5O56O8U8CJVV4KQNSO5AEMVJF66Q9ASUAAJG): User: arn:aws:= sts::980678866538:assumed-role/stevel-s3-restricted/stevel is not authorize= d to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:eu-west-1:= 980678866538:table/hwdev-steve-ireland-new (Service: AmazonDynamoDBv2; Stat= us Code: 400; Error Code: AccessDeniedException; Request ID: 2I6CMPEON2V56J= KK5O56O8U8CJVV4KQNSO5AEMVJF66Q9ASUAAJG) =09at org.apache.hadoop.fs.s3a.S3AUtils.translateDynamoDBException(S3AUtils= .java:389) =09at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:18= 1) =09at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111) =09at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260) =09at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317) =09at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256) =09at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:188) =09at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:210) =09at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.provisionTable= (DynamoDBMetadataStore.java:1080) =09at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.provisionTable= Blocking(DynamoDBMetadataStore.java:1095) =09at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.updateParamete= rs(DynamoDBMetadataStore.java:1204) =09at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$SetCapacity.run(S3GuardT= ool.java:498) =09at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:350= ) =09at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) =09at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:148= 9) =09at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:14= 98) Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:= User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel i= s not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dyna= modb:eu-west-1:980678866538:table/hwdev-steve-ireland-new (Service: AmazonD= ynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID:= 2I6CMPEON2V56JKK5O56O8U8CJVV4KQNSO5AEMVJF66Q9ASUAAJG) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorRespon= se(AmazonHttpClient.java:1639) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest= (AmazonHttpClient.java:1304) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(Ama= zonHttpClient.java:1056) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonH= ttpClient.java:743) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(= AmazonHttpClient.java:717) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHtt= pClient.java:699) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(Amazon= HttpClient.java:667) =09at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execu= te(AmazonHttpClient.java:649) =09at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513= ) =09at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(Amazo= nDynamoDBClient.java:2925) =09at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonD= ynamoDBClient.java:2901) =09at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeUpdateT= able(AmazonDynamoDBClient.java:2757) =09at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateTable(Am= azonDynamoDBClient.java:2733) =09at com.amazonaws.services.dynamodbv2.document.Table.updateTable(Table.ja= va:371) =09at com.amazonaws.services.dynamodbv2.document.Table.updateTable(Table.ja= va:463) =09at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.lambda$provisi= onTable$7(DynamoDBMetadataStore.java:1083) =09at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$2(Invoker.java:190) =09at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109) =09... 13 more 2018-06-29 12:31:59,988 [main] INFO util.ExitUtil (ExitUtil.java:terminate= (210)) - Exiting with status -1: org.apache.hadoop.fs.s3a.AWSServiceIOExcep= tion: ProvisionTable on hwdev-steve-ireland-new: com.amazonaws.services.dyn= amodbv2.model.AmazonDynamoDBException: User: arn:aws:sts::980678866538:assu= med-role/stevel-s3-restricted/stevel is not authorized to perform: dynamodb= :UpdateTable on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwd= ev-steve-ireland-new (Service: AmazonDynamoDBv2; Status Code: 400; Error Co= de: AccessDeniedException; Request ID: 2I6CMPEON2V56JKK5O56O8U8CJVV4KQNSO5A= EMVJF66Q9ASUAAJG): User: arn:aws:sts::980678866538:assumed-role/stevel-s3-r= estricted/stevel is not authorized to perform: dynamodb:UpdateTable on reso= urce: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new= (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedExce= ption; Request ID: 2I6CMPEON2V56JKK5O56O8U8CJVV4KQNSO5AEMVJF66Q9ASUAAJG) 2018-06-29 12:31:59,990 [pool-2-thread-1] DEBUG s3a.S3AFileSystem (S3AFileS= ystem.java:close(2465)) - Filesystem s3a://hwdev-steve-ireland-new is close= d {code} > s3guard set-capacity to not retry on an access denied exception > --------------------------------------------------------------- > > Key: HADOOP-15573 > URL: https://issues.apache.org/jira/browse/HADOOP-15573 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Reporter: Steve Loughran > Priority: Minor > > when you call {{hadoop s3guard set-capacity}} with restricted access, you= are (correctly) blocked by AWS, but the client keeps retrying. It should f= ail fast on a 400/AccessDenied -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-issues-help@hadoop.apache.org