hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15038) Abstract MetadataStore in S3Guard into a common module.
Date Tue, 14 Nov 2017 23:01:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16252646#comment-16252646

Aaron Fabbri commented on HADOOP-15038:

I am happy to move {{MetadataStore}} and related interfaces outside of {{hadoop-aws}} as soon
as another module needs it. I originally intended for this to be possible, just didn't want
to pollute public hadoop common until it was needed by another client. I know some of the
ADLS developers were interested in using it so I worked to keep it separate.

Can you comment on when you will need this?  It might be a good idea to use HADOOP-14098 as
a top-level  "umbrella" JIRA for your effort, and create subtasks so we can see more details
on how you plan to do the work?

> Abstract MetadataStore in S3Guard into a common module.
> -------------------------------------------------------
>                 Key: HADOOP-15038
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15038
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 3.0.0-beta1
>            Reporter: Genmao Yu
> Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}} into a common
> Based on this work, other filesystem or object store can implement their own metastore
for optimization (known issues like consistency problem and metadata operation performance).
[~stevel@apache.org] and other guys have done many base and great works in {{S3Guard}}. It
is very helpful to start work. I did some perf test in HADOOP-14098, and started related work
for Aliyun OSS.  Indeed there are still works to do for {{S3Guard}}, like metadata cache inconsistent
with S3 and so on. It also will be a problem for other object store. However, we can do these
works in parallel.
> [~stevel@apache.org] [~fabbri] [~drankye] Any suggestion is appreciated.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message