Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 19 Oct 2017 17:55:00 +0000 (UTC)
From: "Aaron Fabbri (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.13044535.1487591813000.10057.1508435700644@Atlassian.JIRA>
In-Reply-To: <JIRA.13044535.1487591813000@Atlassian.JIRA>
References: <JIRA.13044535.1487591813000@Atlassian.JIRA> <JIRA.13044535.1487591813660@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HADOOP-14098) AliyunOSS: improve the
 performance of object metadata operation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 19 Oct 2017 17:55:07 -0000


    [ https://issues.apache.org/jira/browse/HADOOP-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211442#comment-16211442 ] 

Aaron Fabbri commented on HADOOP-14098:
---------------------------------------

I made efforts to keep the MetadataStore part of S3Guard a separate layer that other filesystems could use.

In S3Guard, we use a MetadataStore as a trailing log of metadata edits used to guard against list/stat inconsistency. The MetadataStore interface is also designed to be used as a limited-lifetime cache of FileStatus objects which is demand-loaded and does not need to contain metadata for all files in the underlying FS.  (e.g. implementations may only cache recently-seen entries).

Some pointers to get started:

1. The [MetadataStore|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStore.java] interface.  Note there are zero imports of S3A code there.  It does live in hadoop-tools/hadoop-aws, but I expected us to move it to a common place as soon as another FileSystem uses it.  (Note, there used to be S3A specific code for empty directory behavior, but those have been fixed and removed from MetadataStore layer).

2. A local (in-memory) implementation of the interface is [LocalMetadataStore|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java].  This is not for production use at this time: The goal is to have an easy to run reference implementation for tests.  It is not a perfect implementation but it is small (<500 LOC) and supports authoritative directory listing bit (which the Dynamo implementation does not yet support).  You could use this as a test implementation for integrating with a FileSystem.

3. There is a nice set of contract tests that validate that an implementation (i.e. different back end) of MetadataStore works correctly.  The base test class is [MetadataStoreTestBase|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java].  If you wanted to develop a new back end you could essentially use this for test-driven development.

I agree with Steve L that this is a big amount of work.  A major caveat of the approach is the lack of transactions around updating the two sources of truth (the FS and the MetadataStore).  This means things can get out of sync when failures happen. The cost of transactions is prohibitive with the backend we use (Dynamo), so our strategy in dealing with this is (1) use soft state (entries in MetadataStore are expired via a prune CLI command that is scheduled).  (2) Have good CLI tools for detecting and fixing any inconsistencies.  Another feature of the design is that it is always safe to delete all the data in the MetadataStore.. Think of it as a cache flush that can be used to clear inconsistencies in the worst case.  (Also: Deleting *some* of the data may or may not be safe depending on the implementation of the MetadataStore).


> AliyunOSS: improve the performance of object metadata operation
> ---------------------------------------------------------------
>
>                 Key: HADOOP-14098
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14098
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Genmao Yu
>            Assignee: Genmao Yu
>
> Open this JIRA to research and address the potential request performance issue.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org