hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mingliang Liu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HADOOP-13908) S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore
Date Fri, 06 Jan 2017 09:25:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15804093#comment-15804093
] 

Mingliang Liu edited comment on HADOOP-13908 at 1/6/17 9:25 AM:
----------------------------------------------------------------

When I tested the v4 patch when the table was not existent (and auto-creation is enabled),
I still got "requested resource not found" exception, which is not expected. There is some
offline discussion with Chris and Steve, and I'll post the progress or new patch later.

{code}
$ mvn -Dtest=none -Dit.test='ITestS3A*' -q -Dscale clean verify                          
              

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
Tests run: 10, Failures: 0, Errors: 10, Skipped: 0, Time elapsed: 7.569 sec <<< FAILURE!
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
testOverwriteNonEmptyDirectory(org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate) 
Time elapsed: 7.008 sec  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSServiceIOException: get on s3a://mliu-s3guard/test: com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException:
Requested resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException;
Request ID: ooxx): Requested resource not found (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ResourceNotFoundException; Request ID: ooxx)
	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:171)
	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:94)
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.get(DynamoDBMetadataStore.java:309)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1607)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1551)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1520)
	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2218)
	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338)
	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:193)
...
{code}


was (Author: liuml07):
When I tested the v4 patch when the table was not existent (and auto-creation is enabled),
I still got "requested resource not found" exception, which is not expected. There is some
offline discussion on-going, and I'll post the progress or new patch later.

{code}
$ mvn -Dtest=none -Dit.test='ITestS3A*' -q -Dscale clean verify                          
              

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
Tests run: 10, Failures: 0, Errors: 10, Skipped: 0, Time elapsed: 7.569 sec <<< FAILURE!
- in org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate
testOverwriteNonEmptyDirectory(org.apache.hadoop.fs.contract.s3a.ITestS3AContractCreate) 
Time elapsed: 7.008 sec  <<< ERROR!
org.apache.hadoop.fs.s3a.AWSServiceIOException: get on s3a://mliu-s3guard/test: com.amazonaws.services.dynamodbv2.model.ResourceNotFoundException:
Requested resource not found (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ResourceNotFoundException;
Request ID: ooxx): Requested resource not found (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ResourceNotFoundException; Request ID: ooxx)
	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:171)
	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:94)
	at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.get(DynamoDBMetadataStore.java:309)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1607)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:1551)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:1520)
	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2218)
	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338)
	at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:193)
...
{code}

> S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore
> ----------------------------------------------------------------------------------
>
>                 Key: HADOOP-13908
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13908
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>         Attachments: HADOOP-13908-HADOOP-13345.000.patch, HADOOP-13908-HADOOP-13345.001.patch,
HADOOP-13908-HADOOP-13345.002.patch, HADOOP-13908-HADOOP-13345.002.patch, HADOOP-13908-HADOOP-13345.003.patch,
HADOOP-13908-HADOOP-13345.004.patch
>
>
> This was based on discussion in [HADOOP-13455]. Though we should not create table unless
the config {{fs.s3a.s3guard.ddb.table.create}} is set true, we still have to get the existing
table in {{DynamoDBMetadataStore#initialize()}} and wait for its becoming active, before any
table/item operations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message