hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14036) S3Guard: intermittent duplicate item keys failure
Date Thu, 16 Mar 2017 21:58:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15928983#comment-15928983
] 

Mingliang Liu commented on HADOOP-14036:
----------------------------------------

# The base test case should cover both empty and nonempty directories. All subclasses *should*
pass.
# As individual modules need special settings for integration tests (e.g. credentials), we
can track elsewhere and fix them _if they fail_
# Alternatively I can move the newly added test to S3A of course. That will be safer as a
compromise.

What I tested: S3N works; S3A works; Aliyun OSS works; HDFS works; WebHDFS works; RawLocalFS
works.

I don't have sponsored credentials of ADL/Azure/OpenStack credentials. ADL has a $100/mo free
program for Apache committers and my petition is still pending.

> S3Guard: intermittent duplicate item keys failure
> -------------------------------------------------
>
>                 Key: HADOOP-14036
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14036
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Aaron Fabbri
>            Assignee: Mingliang Liu
>         Attachments: HADOOP-14036-HADOOP-13345.000.patch, HADOOP-14036-HADOOP-13345.001.patch,
HADOOP-14036-HADOOP-13345.002.patch
>
>
> I see this occasionally when running integration tests with -Ds3guard -Ddynamo:
> {noformat}
> testRenameToDirWithSamePrefixAllowed(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract)
 Time elapsed: 2.756 sec  <<< ERROR!
> org.apache.hadoop.fs.s3a.AWSServiceIOException: move: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:
Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ValidationException; Request ID: QSBVQV69279UGOB4AJ4NO9Q86VVV4KQNSO5AEMVJF66Q9ASUAAJG):
Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ValidationException; Request ID: QSBVQV69279UGOB4AJ4NO9Q86VVV4KQNSO5AEMVJF66Q9ASUAAJG)
>         at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:178)
>         at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.move(DynamoDBMetadataStore.java:408)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:869)
>         at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:662)
>         at org.apache.hadoop.fs.FileSystemContractBaseTest.rename(FileSystemContractBaseTest.java:525)
>         at org.apache.hadoop.fs.FileSystemContractBaseTest.testRenameToDirWithSamePrefixAllowed(FileSystemContractBaseTest.java:669)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcces
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message