hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.
Date Sat, 03 Dec 2016 00:52:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15717021#comment-15717021

Aaron Fabbri commented on HADOOP-13449:

I did a little research on #3. It looks like you cannot do a prefix scan on a partition key
for DynamoDB.  This seems to imply that, considering an operation {{deleteSubtree(delete_path)}},
a simple search by prefix to find all entries with paths that begin with {{delete_path}} would
actually be a full table scan.  If I'm right, that is unfortunate.

The problem with the existing deleteSubtree(delete_path) implementation is that all the children
under delete_path might not be reachable from delete_path by doing a simple tree walk over
the state in the MetadataStore.  The algorithm would work, however, if, when we created a
file, we also created all its ancestor directories up to the root.  This would establish an
invariant that

For any path p in DDB MetadataStore
For each ancestor a_i from p to the root
a_i is in DDB MetadataStore

This actually sounds reasonable.  Can we do it without changing the {{MetadataStore}} interface?
 I think we can: when we create(path), we always have the full absolute 'path', so we know
the names of the ancestors all the way to the root.


> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>                 Key: HADOOP-13449
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13449
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Mingliang Liu
>         Attachments: HADOOP-13449-HADOOP-13345.000.patch, HADOOP-13449-HADOOP-13345.001.patch,
HADOOP-13449-HADOOP-13345.002.patch, HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch,
HADOOP-13449-HADOOP-13345.005.patch, HADOOP-13449-HADOOP-13345.006.patch, HADOOP-13449-HADOOP-13345.007.patch,
HADOOP-13449-HADOOP-13345.008.patch, HADOOP-13449-HADOOP-13345.009.patch, HADOOP-13449-HADOOP-13345.010.patch
> Provide an implementation of the metadata store backed by DynamoDB.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message