hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15573) s3guard set-capacity to not retry on an access denied exception
Date Fri, 29 Jun 2018 11:36:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527493#comment-16527493
] 

Steve Loughran commented on HADOOP-15573:
-----------------------------------------

{code}
bin/hadoop s3guard $ARN set-capacity -read 10 $IRL
2018-06-29 12:31:45,179 [main] DEBUG s3guard.S3GuardTool (S3GuardTool.java:run(1458)) - Executing
command set-capacity
Read capacity set to 10
2018-06-29 12:31:45,207 [main] DEBUG s3guard.S3GuardTool (S3GuardTool.java:initS3AFileSystem(301))
- updated bucket store option org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
2018-06-29 12:31:45,974 [main] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:initialize(237))
- Initializing S3AFileSystem for hwdev-steve-ireland-new
2018-06-29 12:31:45,976 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1001))
- Propagating entries under fs.s3a.bucket.hwdev-steve-ireland-new.
2018-06-29 12:31:46,055 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1022))
- Updating fs.s3a.endpoint from [core-site.xml]
2018-06-29 12:31:46,055 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1022))
- Updating fs.s3a.committer.magic.enabled from [core-site.xml]
2018-06-29 12:31:46,056 [main] DEBUG s3a.S3AUtils (S3AUtils.java:propagateBucketOptions(1022))
- Updating fs.s3a.metadatastore.impl from [S3AUtils]
2018-06-29 12:31:46,260 [main] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(628))
- Credential provider class is org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
2018-06-29 12:31:46,261 [main] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(628))
- Credential provider class is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
2018-06-29 12:31:46,269 [main] DEBUG auth.AssumedRoleCredentialProvider (AssumedRoleCredentialProvider.java:<init>(117))
- AssumedRoleCredentialProvider{role='arn:aws:iam::980678866538:role/stevel-s3-restricted',
session'stevel', duration=1800}
2018-06-29 12:31:46,271 [main] DEBUG auth.AssumedRoleCredentialProvider (AssumedRoleCredentialProvider.java:<init>(130))
- Credentials to obtain role credentials: AWSCredentialProviderList: SimpleAWSCredentialsProvider
2018-06-29 12:31:46,827 [main] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:getCredentials(122))
- Using credentials from SimpleAWSCredentialsProvider
2018-06-29 12:31:47,566 [main] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProviderSet(572))
- For URI s3a://hwdev-steve-ireland-new//, using credentials AWSCredentialProviderList: AssumedRoleCredentialProvider{role='arn:aws:iam::980678866538:role/stevel-s3-restricted',
session'stevel', duration=1800}
2018-06-29 12:31:47,567 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.connection.maximum is 15
2018-06-29 12:31:47,567 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.attempts.maximum is 20
2018-06-29 12:31:47,567 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.connection.establish.timeout is 5000
2018-06-29 12:31:47,568 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.connection.timeout is 5000
2018-06-29 12:31:47,568 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.socket.send.buffer is 65536
2018-06-29 12:31:47,568 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.socket.recv.buffer is 32678
2018-06-29 12:31:47,570 [main] DEBUG s3a.S3AFileSystem (DefaultS3ClientFactory.java:initUserAgent(183))
- Using User-Agent: Hadoop 3.2.0-SNAPSHOT
2018-06-29 12:31:47,637 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.paging.maximum is 5000
2018-06-29 12:31:47,640 [main] DEBUG s3a.S3AUtils (S3AUtils.java:longBytesOption(887)) - Value
of fs.s3a.block.size is 33554432
2018-06-29 12:31:47,640 [main] DEBUG s3a.S3AUtils (S3AUtils.java:longBytesOption(887)) - Value
of fs.s3a.readahead.range is 524288
2018-06-29 12:31:47,640 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.max.total.tasks is 5
2018-06-29 12:31:47,641 [main] DEBUG s3a.S3AUtils (S3AUtils.java:longOption(865)) - Value
of fs.s3a.threads.keepalivetime is 60
2018-06-29 12:31:47,656 [main] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:getCredentials(122))
- Using credentials from AssumedRoleCredentialProvider{role='arn:aws:iam::980678866538:role/stevel-s3-restricted',
session'stevel', duration=1800}
2018-06-29 12:31:47,836 [main] DEBUG s3a.S3AUtils (S3AUtils.java:getEncryptionAlgorithm(1253))
- Data is unencrypted
2018-06-29 12:31:47,837 [main] DEBUG s3a.S3AUtils (S3AUtils.java:getEncryptionAlgorithm(1256))
- Using SSE-C with empty key
2018-06-29 12:31:47,837 [main] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:initialize(313))
- Input fadvise policy = normal
2018-06-29 12:31:47,837 [main] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:initialize(317))
- Filesystem support for magic committers is enabled
2018-06-29 12:31:47,843 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.fast.upload.active.blocks is 4
2018-06-29 12:31:47,843 [main] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:initialize(333))
- Using S3ABlockOutputStream with buffer = disk; block=8388608; queue limit=4
2018-06-29 12:31:47,846 [main] DEBUG s3guard.S3Guard (S3Guard.java:getMetadataStoreClass(125))
- Metastore option source fs.s3a.bucket.hwdev-steve-ireland-new.metadatastore.impl via [S3AUtils]
2018-06-29 12:31:47,847 [main] DEBUG s3guard.S3Guard (S3Guard.java:getMetadataStore(97)) -
Using NullMetadataStore metadata store for s3a filesystem
2018-06-29 12:31:47,847 [main] DEBUG s3a.S3AUtils (S3AUtils.java:longOption(865)) - Value
of fs.s3a.multipart.purge.age is 3600000
2018-06-29 12:31:47,924 [main] DEBUG s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:initialize(277))
- Inferring DynamoDB region from S3 bucket: eu-west-1
2018-06-29 12:31:47,924 [main] DEBUG s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:createDynamoDB(255))
- Creating DynamoDB client class org.apache.hadoop.fs.s3a.s3guard.DynamoDBClientFactory$DefaultDynamoDBClientFactory
with S3 region eu-west-1
2018-06-29 12:31:47,924 [main] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(628))
- Credential provider class is org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
2018-06-29 12:31:47,924 [main] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProvider(628))
- Credential provider class is org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
2018-06-29 12:31:47,925 [main] DEBUG auth.AssumedRoleCredentialProvider (AssumedRoleCredentialProvider.java:<init>(117))
- AssumedRoleCredentialProvider{role='arn:aws:iam::980678866538:role/stevel-s3-restricted',
session'stevel', duration=1800}
2018-06-29 12:31:47,925 [main] DEBUG auth.AssumedRoleCredentialProvider (AssumedRoleCredentialProvider.java:<init>(130))
- Credentials to obtain role credentials: AWSCredentialProviderList: SimpleAWSCredentialsProvider
2018-06-29 12:31:47,927 [main] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:getCredentials(122))
- Using credentials from SimpleAWSCredentialsProvider
2018-06-29 12:31:48,378 [main] DEBUG s3a.S3AUtils (S3AUtils.java:createAWSCredentialProviderSet(572))
- For URI (null URI), using credentials AWSCredentialProviderList: AssumedRoleCredentialProvider{role='arn:aws:iam::980678866538:role/stevel-s3-restricted',
session'stevel', duration=1800}
2018-06-29 12:31:48,379 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.connection.maximum is 15
2018-06-29 12:31:48,379 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.attempts.maximum is 20
2018-06-29 12:31:48,379 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.connection.establish.timeout is 5000
2018-06-29 12:31:48,379 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.connection.timeout is 5000
2018-06-29 12:31:48,380 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.socket.send.buffer is 65536
2018-06-29 12:31:48,380 [main] DEBUG s3a.S3AUtils (S3AUtils.java:intOption(844)) - Value of
fs.s3a.socket.recv.buffer is 32678
2018-06-29 12:31:48,380 [main] DEBUG s3a.S3AFileSystem (DefaultS3ClientFactory.java:initUserAgent(183))
- Using User-Agent: Hadoop 3.2.0-SNAPSHOT
2018-06-29 12:31:48,382 [main] DEBUG s3guard.DynamoDBClientFactory (DynamoDBClientFactory.java:createDynamoDBClient(84))
- Creating DynamoDB client in region eu-west-1
2018-06-29 12:31:48,472 [main] DEBUG s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:initTable(893))
- Binding to table hwdev-steve-ireland-new
2018-06-29 12:31:48,519 [main] DEBUG s3a.AWSCredentialProviderList (AWSCredentialProviderList.java:getCredentials(122))
- Using credentials from AssumedRoleCredentialProvider{role='arn:aws:iam::980678866538:role/stevel-s3-restricted',
session'stevel', duration=1800}
2018-06-29 12:31:48,867 [main] DEBUG s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:initTable(895))
- Table state: {AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName:
parent,AttributeType: S}],TableName: hwdev-steve-ireland-new,KeySchema: [{AttributeName: parent,KeyType:
HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Fri Jun
29 12:20:03 BST 2018,ProvisionedThroughput: {NumberOfDecreasesToday: 0,ReadCapacityUnits:
500,WriteCapacityUnits: 100},TableSizeBytes: 0,ItemCount: 0,TableArn: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new,TableId:
99dfc220-a142-4816-9b3b-bfed91de2d30,}
2018-06-29 12:31:48,917 [main] DEBUG s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:initTable(920))
- Using existing DynamoDB table hwdev-steve-ireland-new in region eu-west-1 created Fri Jun
29 12:20:13 BST 2018
2018-06-29 12:31:48,918 [main] INFO  s3guard.S3GuardTool (S3GuardTool.java:initMetadataStore(270))
- Metadata store DynamoDBMetadataStore{region=eu-west-1, tableName=hwdev-steve-ireland-new}
is initialized.
2018-06-29 12:31:48,954 [main] INFO  s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:updateParameters(1200))
- Current table capacity is read: 500, write: 100
2018-06-29 12:31:48,954 [main] INFO  s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:updateParameters(1202))
- Changing capacity of table to read: 10, write: 100
2018-06-29 12:31:49,041 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  1
2018-06-29 12:31:49,546 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#1
2018-06-29 12:31:49,588 [main] INFO  s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:retryEvent(1251))
- Retrying ProvisionTable on hwdev-steve-ireland-new: org.apache.hadoop.fs.s3a.AWSServiceIOException:
ProvisionTable on hwdev-steve-ireland-new: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:
User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel is not authorized
to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID:
KHFS1RER89LU8OAC6967B5U8S7VV4KQNSO5AEMVJF66Q9ASUAAJG): User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel
is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID:
KHFS1RER89LU8OAC6967B5U8S7VV4KQNSO5AEMVJF66Q9ASUAAJG)
2018-06-29 12:31:49,589 [main] DEBUG s3guard.DynamoDBMetadataStore (DynamoDBMetadataStore.java:retryEvent(1252))
- Retrying ProvisionTable on hwdev-steve-ireland-new
org.apache.hadoop.fs.s3a.AWSServiceIOException: ProvisionTable on hwdev-steve-ireland-new:
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel
is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID:
KHFS1RER89LU8OAC6967B5U8S7VV4KQNSO5AEMVJF66Q9ASUAAJG): User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel
is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID:
KHFS1RER89LU8OAC6967B5U8S7VV4KQNSO5AEMVJF66Q9ASUAAJG)
	at org.apache.hadoop.fs.s3a.S3AUtils.translateDynamoDBException(S3AUtils.java:389)
	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:181)
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260)
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317)
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256)
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:188)
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:210)
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.provisionTable(DynamoDBMetadataStore.java:1080)
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.provisionTableBlocking(DynamoDBMetadataStore.java:1095)
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.updateParameters(DynamoDBMetadataStore.java:1204)
	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$SetCapacity.run(S3GuardTool.java:498)
	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:350)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1489)
	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1498)
Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel
is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID:
KHFS1RER89LU8OAC6967B5U8S7VV4KQNSO5AEMVJF66Q9ASUAAJG)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2925)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2901)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeUpdateTable(AmazonDynamoDBClient.java:2757)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateTable(AmazonDynamoDBClient.java:2733)
	at com.amazonaws.services.dynamodbv2.document.Table.updateTable(Table.java:371)
	at com.amazonaws.services.dynamodbv2.document.Table.updateTable(Table.java:463)
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.lambda$provisionTable$7(DynamoDBMetadataStore.java:1083)
	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$2(Invoker.java:190)
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
	... 13 more
2018-06-29 12:31:49,591 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  2
2018-06-29 12:31:50,093 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#2
2018-06-29 12:31:50,136 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  3
2018-06-29 12:31:50,638 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#3
2018-06-29 12:31:50,684 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  4
2018-06-29 12:31:51,184 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#4
2018-06-29 12:31:51,228 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  5
2018-06-29 12:31:51,730 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#5
2018-06-29 12:31:51,774 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  6
2018-06-29 12:31:52,276 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#6
2018-06-29 12:31:52,319 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  7
2018-06-29 12:31:52,822 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#7
2018-06-29 12:31:52,865 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  8
2018-06-29 12:31:53,367 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#8
2018-06-29 12:31:53,411 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  9
2018-06-29 12:31:53,915 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#9
2018-06-29 12:31:53,959 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  10
2018-06-29 12:31:54,461 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#10
2018-06-29 12:31:54,505 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  11
2018-06-29 12:31:55,009 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#11
2018-06-29 12:31:55,052 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  12
2018-06-29 12:31:55,556 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#12
2018-06-29 12:31:55,605 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  13
2018-06-29 12:31:56,109 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#13
2018-06-29 12:31:56,153 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  14
2018-06-29 12:31:56,658 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#14
2018-06-29 12:31:56,701 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  15
2018-06-29 12:31:57,202 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#15
2018-06-29 12:31:57,245 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  16
2018-06-29 12:31:57,750 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#16
2018-06-29 12:31:57,792 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  17
2018-06-29 12:31:58,298 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#17
2018-06-29 12:31:58,342 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  18
2018-06-29 12:31:58,847 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#18
2018-06-29 12:31:58,889 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  19
2018-06-29 12:31:59,394 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#19
2018-06-29 12:31:59,438 [main] DEBUG s3a.S3AStorageStatistics (S3AStorageStatistics.java:incrementCounter(63))
- ignored_errors += 1  ->  20
2018-06-29 12:31:59,943 [main] DEBUG s3a.Invoker (Invoker.java:retryUntranslated(314)) - retry
#20
org.apache.hadoop.fs.s3a.AWSServiceIOException: ProvisionTable on hwdev-steve-ireland-new:
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel
is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID:
2I6CMPEON2V56JKK5O56O8U8CJVV4KQNSO5AEMVJF66Q9ASUAAJG): User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel
is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID:
2I6CMPEON2V56JKK5O56O8U8CJVV4KQNSO5AEMVJF66Q9ASUAAJG)
	at org.apache.hadoop.fs.s3a.S3AUtils.translateDynamoDBException(S3AUtils.java:389)
	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:181)
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:260)
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:317)
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:256)
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:188)
	at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:210)
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.provisionTable(DynamoDBMetadataStore.java:1080)
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.provisionTableBlocking(DynamoDBMetadataStore.java:1095)
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.updateParameters(DynamoDBMetadataStore.java:1204)
	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$SetCapacity.run(S3GuardTool.java:498)
	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:350)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1489)
	at org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1498)
Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel
is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID:
2I6CMPEON2V56JKK5O56O8U8CJVV4KQNSO5AEMVJF66Q9ASUAAJG)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2925)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2901)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeUpdateTable(AmazonDynamoDBClient.java:2757)
	at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateTable(AmazonDynamoDBClient.java:2733)
	at com.amazonaws.services.dynamodbv2.document.Table.updateTable(Table.java:371)
	at com.amazonaws.services.dynamodbv2.document.Table.updateTable(Table.java:463)
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.lambda$provisionTable$7(DynamoDBMetadataStore.java:1083)
	at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$2(Invoker.java:190)
	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
	... 13 more
2018-06-29 12:31:59,988 [main] INFO  util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting
with status -1: org.apache.hadoop.fs.s3a.AWSServiceIOException: ProvisionTable on hwdev-steve-ireland-new:
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel
is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID:
2I6CMPEON2V56JKK5O56O8U8CJVV4KQNSO5AEMVJF66Q9ASUAAJG): User: arn:aws:sts::980678866538:assumed-role/stevel-s3-restricted/stevel
is not authorized to perform: dynamodb:UpdateTable on resource: arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request ID:
2I6CMPEON2V56JKK5O56O8U8CJVV4KQNSO5AEMVJF66Q9ASUAAJG)
2018-06-29 12:31:59,990 [pool-2-thread-1] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:close(2465))
- Filesystem s3a://hwdev-steve-ireland-new is closed
{code}

> s3guard set-capacity to not retry on an access denied exception
> ---------------------------------------------------------------
>
>                 Key: HADOOP-15573
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15573
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Steve Loughran
>            Priority: Minor
>
> when you call {{hadoop s3guard set-capacity}} with restricted access, you are (correctly)
blocked by AWS, but the client keeps retrying. It should fail fast on a 400/AccessDenied



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message