Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 28 Mar 2017 22:52:41 +0000 (UTC)
From: "Aaron Fabbri (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13058767.1490313556000.148189.1490741561677@Atlassian.JIRA>
In-Reply-To: <JIRA.13058767.1490313556000@Atlassian.JIRA>
References: <JIRA.13058767.1490313556000@Atlassian.JIRA> <JIRA.13058767.1490313556533@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-14226) S3Guard:
 ITestDynamoDBMetadataStoreScale is not cleaning up test data
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 28 Mar 2017 22:52:51 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-14226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946121#comment-15946121 ] 

Aaron Fabbri commented on HADOOP-14226:
---------------------------------------

Hi [~liuml07].  I was thinking about this issue, and it seems that DyamoDBMetadataStore should make sure that is has entries for the ancestors of any path.  This is an implementation detail of that particular MetadataStore.  I was thinking that {{DynamoDBMetadataStore#move()}} needs to do the same thing that {{put()}} does: use a function like {{DynamoDBMetadataStore#fullPathsToPut()}} to make sure it has records for each directory up to the root.

This is different than HADOOP-14236, where S3A FS was failing to be consistent in the way it is calling the MetadataStore.  In that bug, the fix only supplies intermediate paths that are "inferred" in S3.   It correctly does not continue adding paths all the way up to the root.  So I think move() also needs to add the directory entries from the root down to the directory being moved.


> S3Guard: ITestDynamoDBMetadataStoreScale is not cleaning up test data
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-14226
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14226
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: HADOOP-13345
>            Reporter: Mingliang Liu
>            Assignee: Mingliang Liu
>            Priority: Minor
>         Attachments: HADOOP-14226-HADOOP-13345.000.patch
>
>
> After running {{ITestDynamoDBMetadataStoreScale}}, the test data is not cleaned up. There is a call to {{clearMetadataStore(ms, count);}} in the finally clause though. The reason is that, the internally called method {{DynamoDBMetadataStore::deleteSubtree()}} is assuming there should be an item for the parent dest path:
> {code}
> parent=/fake-bucket, child=moved-here, is_dir=true
> {code}
> In DynamoDBMetadataStore implementation, we assume that _if a path exists, all its ancestors will also exist in the table_. We need to pre-create dest path to maintain this invariant so that test data can be cleaned up successfully.
> I think there may be other tests with the same problem. Let's identify/address them separately.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org