hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14036) S3Guard: intermittent duplicate item keys failure
Date Sat, 28 Jan 2017 13:29:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15844053#comment-15844053
] 

Steve Loughran commented on HADOOP-14036:
-----------------------------------------

we've seen this intermittently before, not worked out what is happening. 

We think that the list of keys for delete being built up has duplicates in, but aren't sure
where that's coming from.

> S3Guard: intermittent duplicate item keys failure
> -------------------------------------------------
>
>                 Key: HADOOP-14036
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14036
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Aaron Fabbri
>
> I see this occasionally when running integration tests with -Ds3guard -Ddynamo:
> {noformat}
> testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)
 Time elapsed: 2.756 sec  <<< ERROR!
> org.apache.hadoop.fs.s3a.AWSServiceIOException: move: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:
Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ValidationException; Request ID: QSBVQV69279UGOB4AJ4NO9Q86VVV4KQNSO5AEMVJF66Q9ASUAAJG):
Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ValidationException; Request ID: QSBVQV69279UGOB4AJ4NO9Q86VVV4KQNSO5AEMVJF66Q9ASUAAJG)
>         at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178)
>         at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:408)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:869)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:662)
>         at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:525)
>         at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:669)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcces
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message