hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Genmao Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14098) AliyunOSS: improve the performance of object metadata operation
Date Fri, 10 Nov 2017 12:09:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247399#comment-16247399
] 

Genmao Yu commented on HADOOP-14098:
------------------------------------

h3. AliyunOSS FS + metastore POC

1. metastore: a common module of {{MetadataStore}} interface to manager metadata of OSS object.
Other filesystem or object store can manager their metadata based on specific implementation
of metastore
2. In this POC, I implemented one common metastore based on HDFS, i.e. we use HDFS to manager
metadata of object store. One advantage is the semantic consistency, we can use HDFS backened
metastore to manage metadata just like use HDFS. Another advantage is this HDFS based metastore
is platform independent, i.e. we can use it to manager any object store metadata.

h4. Goals:
  - Test if metastore can accelerate the performance of metadata operation.

h4. Non-Goals:
  - This test is not a pressure test of cluster. The test result does not indicate any platform
or product performance.

| operation|unit|oss fs + metastore|oss fs|hdfs|
|CreateOp| | | | |
| |operations/sec|0.243|0.252|1.84|
| |successes/sec|0.152|0.161|0.324|
|DeleteOp| | | | |
| |operations/sec|52.479|43.499|328.515|
| |successes/sec|19.438|16.947|162.943|
|ListOp| | | | |
| |directory entries/sec|303.461|51.949|386.441|
| |operations/sec|250.05|37.191|235.405|
| |successes/sec|247.85|37.043|233.898|
|MkdirOp| | | | |
| |operations/sec|227.853|73.738|225.388|
| |successes/sec|227.853|73.738|225.388|
|ReadOp| | | | |
| |operations/sec|2.039|1.606|47.426|
| |successes/sec|0.127|0.132|6.222|
|RenameOp| | | | |
| |operations/sec|2.542|2.986|578.972|
| |successes/sec|0.578|0.507|163.965|
|AppendOp| |No Support|No Support| |

With metastore, we can find:
- {{ListOp}} and {{MkdirOp}} have a similar performance with HDFS.
- Others like {{RenameOp}} {{ReadOp}} {{DeleteOp}} still are much worse than HDFS.

> AliyunOSS: improve the performance of object metadata operation
> ---------------------------------------------------------------
>
>                 Key: HADOOP-14098
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14098
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Genmao Yu
>            Assignee: Genmao Yu
>
> Open this JIRA to research and address the potential request performance issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message