hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13651) S3Guard: S3AFileSystem Integration with MetadataStore
Date Thu, 27 Oct 2016 00:40:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610185#comment-15610185

Lei (Eddy) Xu commented on HADOOP-13651:

Hi, [~liuml07] and [~fabbri]

[~fabbri] and I had an offline discussion today, the followings are my notes

* We agree that the atomic of creation / deletion / rename (between metadata store and s3)
are difficult  to achieve, as S3 is always visible to some of the clients. We need more thoughts
on these issues. My option is making sure namespace consistency first, but there are other
understandable concerns. 
* For {{rename}}, it should not expect that {{rename(recursive=true)}} can be atomic. 
* {{LocalMetadataStore}} should not use {{LRU/MRU}}, which can not provide consistency due
to that the metadata will be evicted. 
* The current test suites (unit tests and integration tests) can not reliability test eventual
consistency. New tests cases are necessary before merging this branch to branch.
* This patch is large and many of the following works are depended to this patch. My CLI patch
(HADOOP-13650) and DynamoDB metadata store (HADOOP-13449) for instance. 

As this is a feature branch, I'd give a pending +1 once these concerns are appropriately documented
(i.e., adding {{TODO}}).  Thanks.

> S3Guard: S3AFileSystem Integration with MetadataStore
> -----------------------------------------------------
>                 Key: HADOOP-13651
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13651
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
>         Attachments: HADOOP-13651-HADOOP-13345.001.patch, HADOOP-13651-HADOOP-13345.002.patch
> Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata consistency
and caching.
> Implementation should have minimal overhead when no MetadataStore is configured.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message