hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
Date Wed, 29 Oct 2014 17:13:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188595#comment-14188595

Zhijie Shen commented on YARN-2765:

bq. This should work if the leveldb database is on a network store like a filer.

Thanks for sharing. This is an interesting use case that I'm not aware of before.

bq. I briefly considered using rocksdb for this but decided against it for a couple of reasons:

It's not particularly related to this Jira, but I just want to think it out loudly. It seems
that rocksdb claims to have better performance in terms of I/O than leveldb, while their APIs
are very similar to each other. After we have the leveldb impl, it shouldn't be that difficult
to make a rocksdb impl. Probably leveldb is enough to serve as the state store for RM/NM/JHS,
but the timeline server may want a stronger one. Rocksdb may be a compromise before migrating
to fully distributed storage solution based on HBase. And one other merit I've heard about
rocksdb is that it can ride on HDFS. Correct me if I'm wrong, but it seems that rocksdb can
also help to scale out the storage problem as well as support RM HA deployment in a shared
nothing environment (e.g. without a network storage).

I'm not saying we should go with rocksdb now instead of leveldb, as we know it has been used
for other components already. I'm trying to propose if we can think of rocksdb, which looks
stronger but still reasonably simple alternate.

> Add leveldb-based implementation for RMStateStore
> -------------------------------------------------
>                 Key: YARN-2765
>                 URL: https://issues.apache.org/jira/browse/YARN-2765
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-2765.patch, YARN-2765v2.patch
> It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb
would provide some benefits over the existing filesystem store such as better support for
atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem.

This message was sent by Atlassian JIRA

View raw message