hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Genmao Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15038) Abstract MetadataStore in S3Guard into a common module.
Date Fri, 17 Nov 2017 09:28:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16256714#comment-16256714
] 

Genmao Yu commented on HADOOP-15038:
------------------------------------

[~fabbri] Sorry for the late response, and thanks for your comments.

bq. I originally intended for this to be possible, just didn't want to pollute public hadoop
common until it was needed by another client.

What about move it to a separate module? like {{hadoop-metastore}} inside {{hadoop-tools}}.


bq. It might be a good idea to use HADOOP-14098 as a top-level "umbrella" JIRA for your effort,
and create subtasks ...

Absolutely yes. Now I did these works offline and made a POC. As soon as I can confirm the
validity of metastore (call it this way first) for performance improving. After all, I want
use metastore to improve performance of metadata operation. There are still many blocks, like
{{rename}}. It is not a metadata operation issue, but can be done with metastore.

bq. Can you comment on when you will need this? 

Maybe,  separating and my work can be done in parallel. Is there any concern? Besides, I can
help you to do separating work.

> Abstract MetadataStore in S3Guard into a common module.
> -------------------------------------------------------
>
>                 Key: HADOOP-15038
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15038
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 3.0.0-beta1
>            Reporter: Genmao Yu
>
> Open this JIRA to discuss if we should move {{MetadataStore}} in {{S3Guard}} into a common
module. 
> Based on this work, other filesystem or object store can implement their own metastore
for optimization (known issues like consistency problem and metadata operation performance).
[~stevel@apache.org] and other guys have done many base and great works in {{S3Guard}}. It
is very helpful to start work. I did some perf test in HADOOP-14098, and started related work
for Aliyun OSS.  Indeed there are still works to do for {{S3Guard}}, like metadata cache inconsistent
with S3 and so on. It also will be a problem for other object store. However, we can do these
works in parallel.
> [~stevel@apache.org] [~fabbri] [~drankye] Any suggestion is appreciated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message