hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.
Date Thu, 20 Oct 2016 22:29:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593227#comment-15593227

Aaron Fabbri commented on HADOOP-13449:

I'm curious what you guys are doing about path normalization.. For LocalMetadataStore, I'm
using Path as the hashtable key.  We already agreed to require all incoming paths to be absolute,
but this is not quite enough to ensure consistency.  Some callers may hand in an absolute
path without scheme and bucket (a.k.a URI host) qualification, and others may hand in a path
to the same object with both of those things.  

I first tried stripping scheme and bucket away (via {Path#getPathWithoutSchemeAndAuthority()}).
 This causes a problem, though, if multiple FileSystem instances share a metadata store singleton.
 In this case, we actually need the host portion of the URI to distinguish between the different
buckets that may be sharing a single MetadataStore.

So, now I'm thinking that MetadataStore implementations, and DirListingMetadata API, need
to do something like fs.qualify(path) for incoming paths.

1. Does this sound right (always using qualified paths as lookup keys)?
2. Is there something less heavyweight we should do besides fs.qualify(path)?  I'd like common
case to be fast.
3. Should MetadataStore impls. and DirListingMetadata be responsible for this path conversion,
or should we assert that the caller already did it?
3.b. If the former, are we ok with DirListingMetadata keeping a reference to an "owning" FileSystem
instance via its constructor?

> S3Guard: Implement DynamoDBMetadataStore.
> -----------------------------------------
>                 Key: HADOOP-13449
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13449
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Mingliang Liu
>         Attachments: HADOOP-13449-HADOOP-13345.wip.patch
> Provide an implementation of the metadata store backed by DynamoDB.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message