hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.
Date Wed, 16 Nov 2016 01:22:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668965#comment-15668965
] 

Aaron Fabbri commented on HADOOP-13449:
---------------------------------------

{quote}
But for listStatus(), my understanding is that we assume the entry per se is there. 
{quote}

The only thing that matters is the semantics or API contract.  As long as you have the behavior
I outlined, it is correct.  

We cannot require client to put(parent) before put(child), since we may run on an existing
bucket where the directory was already created before we started our cluster.  

{quote}Or else, if we query the DDB and no entries having parent as this path, is the directory
nonexistent, or the directory is empty? DDBMetadataStore should return DirListingMetadata
accordingly. Thanks,
{quote}

Ok, I think this is an implementation detail for DynamoDB.  Two ideas.  #1 seems pretty good:

1. Do a prefix scan.. I thought DynamoDB had built-in support for looking up values by key
prefix.  I.e. {{begins_with}}.  When you do a listChildren(parent), you can just query for
{{key begins_with parent}}?

2. Create the parent path when you create the child so you can implement listChildren() properly
(what I did for initial LocalMetadataStore).

Also, can't you use prefix queries to eliminate the whole {{DescendantsIterator}} thing for
recursive delete?

Thanks for the discussion [~liuml07].


> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>
>                 Key: HADOOP-13449
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13449
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Mingliang Liu
>         Attachments: HADOOP-13449-HADOOP-13345.000.patch, HADOOP-13449-HADOOP-13345.001.patch,
HADOOP-13449-HADOOP-13345.002.patch, HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message