hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14266) S3Guard: S3AFileSystem::listFiles() to employ MetadataStore
Date Wed, 19 Apr 2017 01:29:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973859#comment-15973859
] 

Aaron Fabbri commented on HADOOP-14266:
---------------------------------------

The isAuthoritative change looks good to me.

I also noticed a change in v7 patch to add FileStatusAcceptor checking for ProvidedFileStatusIterator.
 Looks like this was missing before?  I'm curious if we should have a test case covering this?
 If so feel free to file separate JIRA.  You might see a javadoc warning on missing {{@param}}
for the acceptor arg you added.

I'm doing more testing in us-west-2.  So far ran non-parallel dynamo test and got one (unrelated
I think) failure:

{noformat}
mvn clean verify  -Dtest=none -Ddynamo -Ds3guard -Dit.test='ITestS3A*,ITestS3G*,ITestMetadata*'
...
testConcurrentTableCreations(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps) 
Time elapsed: 0.504 sec  <<< ERROR!
java.lang.IllegalArgumentException: No DynamoDB table name configured!
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
        at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:261)
        at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps.testConcurrentTableCreations(ITestS3GuardConcurrentOps.java:81)
{noformat}

 I will also try parallel dynamo test, and then with LocalMetadataStore and fs.s3a.metadatastore.authoritative=true.

> S3Guard: S3AFileSystem::listFiles() to employ MetadataStore
> -----------------------------------------------------------
>
>                 Key: HADOOP-14266
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14266
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>         Attachments: HADOOP-14266-HADOOP-13345.000.patch, HADOOP-14266-HADOOP-13345.001.patch,
HADOOP-14266-HADOOP-13345.002.patch, HADOOP-14266-HADOOP-13345.003.patch, HADOOP-14266-HADOOP-13345.003.patch,
HADOOP-14266-HADOOP-13345.004.patch, HADOOP-14266-HADOOP-13345-005.patch, HADOOP-14266-HADOOP-13345.005.patch,
HADOOP-14266-HADOOP-13345.006.patch, HADOOP-14266-HADOOP-13345.007.patch
>
>
> Similar to [HADOOP-13926], this is to track the effort of employing MetadataStore in
{{S3AFileSystem::listFiles()}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message