Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 27 Oct 2016 00:40:59 +0000 (UTC)
From: "Lei (Eddy) Xu (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13007289.1474671254000.95524.1477528859861@Atlassian.JIRA>
In-Reply-To: <JIRA.13007289.1474671254000@Atlassian.JIRA>
References: <JIRA.13007289.1474671254000@Atlassian.JIRA> <JIRA.13007289.1474671254384@arcas>
Subject: [jira] [Commented] (HADOOP-13651) S3Guard: S3AFileSystem
 Integration with MetadataStore
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 27 Oct 2016 00:41:01 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-13651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610185#comment-15610185 ] 

Lei (Eddy) Xu commented on HADOOP-13651:
----------------------------------------

Hi, [~liuml07] and [~fabbri]

[~fabbri] and I had an offline discussion today, the followings are my notes

* We agree that the atomic of creation / deletion / rename (between metadata store and s3) are difficult  to achieve, as S3 is always visible to some of the clients. We need more thoughts on these issues. My option is making sure namespace consistency first, but there are other understandable concerns. 
* For {{rename}}, it should not expect that {{rename(recursive=true)}} can be atomic. 
* {{LocalMetadataStore}} should not use {{LRU/MRU}}, which can not provide consistency due to that the metadata will be evicted. 
* The current test suites (unit tests and integration tests) can not reliability test eventual consistency. New tests cases are necessary before merging this branch to branch.
* This patch is large and many of the following works are depended to this patch. My CLI patch (HADOOP-13650) and DynamoDB metadata store (HADOOP-13449) for instance. 

As this is a feature branch, I'd give a pending +1 once these concerns are appropriately documented (i.e., adding {{TODO}}).  Thanks.


> S3Guard: S3AFileSystem Integration with MetadataStore
> -----------------------------------------------------
>
>                 Key: HADOOP-13651
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13651
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Aaron Fabbri
>            Assignee: Aaron Fabbri
>         Attachments: HADOOP-13651-HADOOP-13345.001.patch, HADOOP-13651-HADOOP-13345.002.patch
>
>
> Modify S3AFileSystem et al. to optionally use a MetadataStore for metadata consistency and caching.
> Implementation should have minimal overhead when no MetadataStore is configured.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org