hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.
Date Wed, 16 Nov 2016 01:22:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668965#comment-15668965

Aaron Fabbri commented on HADOOP-13449:

But for listStatus(), my understanding is that we assume the entry per se is there. 

The only thing that matters is the semantics or API contract.  As long as you have the behavior
I outlined, it is correct.  

We cannot require client to put(parent) before put(child), since we may run on an existing
bucket where the directory was already created before we started our cluster.  

{quote}Or else, if we query the DDB and no entries having parent as this path, is the directory
nonexistent, or the directory is empty? DDBMetadataStore should return DirListingMetadata
accordingly. Thanks,

Ok, I think this is an implementation detail for DynamoDB.  Two ideas.  #1 seems pretty good:

1. Do a prefix scan.. I thought DynamoDB had built-in support for looking up values by key
prefix.  I.e. {{begins_with}}.  When you do a listChildren(parent), you can just query for
{{key begins_with parent}}?

2. Create the parent path when you create the child so you can implement listChildren() properly
(what I did for initial LocalMetadataStore).

Also, can't you use prefix queries to eliminate the whole {{DescendantsIterator}} thing for
recursive delete?

Thanks for the discussion [~liuml07].

> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>                 Key: HADOOP-13449
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13449
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Mingliang Liu
>         Attachments: HADOOP-13449-HADOOP-13345.000.patch, HADOOP-13449-HADOOP-13345.001.patch,
HADOOP-13449-HADOOP-13345.002.patch, HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch
> Provide an implementation of the metadata store backed by DynamoDB.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message