hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14226) S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
Date Tue, 28 Mar 2017 22:52:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946121#comment-15946121

Aaron Fabbri commented on HADOOP-14226:

Hi [~liuml07].  I was thinking about this issue, and it seems that DyamoDBMetadataStore should
make sure that is has entries for the ancestors of any path.  This is an implementation detail
of that particular MetadataStore.  I was thinking that {{DynamoDBMetadataStore#move()}} needs
to do the same thing that {{put()}} does: use a function like {{DynamoDBMetadataStore#fullPathsToPut()}}
to make sure it has records for each directory up to the root.

This is different than HADOOP-14236, where S3A FS was failing to be consistent in the way
it is calling the MetadataStore.  In that bug, the fix only supplies intermediate paths that
are "inferred" in S3.   It correctly does not continue adding paths all the way up to the
root.  So I think move() also needs to add the directory entries from the root down to the
directory being moved.

> S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
> ---------------------------------------------------------------------
>                 Key: HADOOP-14226
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14226
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>            Priority: Minor
>         Attachments: HADOOP-14226-HADOOP-13345.000.patch
> After running {{ITestDynamoDBMetadataStoreScale}}, the test data is not cleaned up. There
is a call to {{clearMetadataStore(ms, count);}} in the finally clause though. The reason is
that, the internally called method {{DynamoDBMetadataStore::deleteSubtree()}} is assuming
there should be an item for the parent dest path:
> {code}
> parent=/fake-bucket, child=moved-here, is_dir=true
> {code}
> In DynamoDBMetadataStore implementation, we assume that _if a path exists, all its ancestors
will also exist in the table_. We need to pre-create dest path to maintain this invariant
so that test data can be cleaned up successfully.
> I think there may be other tests with the same problem. Let's identify/address them separately.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message