hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2765) Add leveldb-based implementation for RMStateStore
Date Wed, 29 Oct 2014 18:32:35 GMT

    [ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188755#comment-14188755

Jason Lowe commented on YARN-2765:

I agree that the timeline server seems like a worthy candidate for rocksdb.  IIUC rocksdb's
main use-case over leveldb is better performance when the database is larger than the node's
RAM, which is likely in the case of the timeline server.

bq. And one other merit I've heard about rocksdb is that it can ride on HDFS.

This is news to me.  I knew rocksdb could be used as a cache of data that came from HDFS or
could be backed-up to HDFS, but I didn't think it could read/write directly to it as part
of normal operations.

bq. There's a rocksdb jni which seems to have windows support: https://github.com/fusesource/rocksdbjni

Awesome, thanks for finding that.  I was looking at the standard org.rocksdb package.  Only
concern with the fusesource option would be if it starts to diverge significantly from the
standard one.  The API is already slightly different between the two, and the fusesource one
hasn't been touched in a year while the org.rocksdb package was updated just last week.

Probably best to continue this conversation in a separate JIRA proposing we consider rocksdb
for the timeline server.  If it works well there it should be very straightforward to provide
store backends for the RM, NM, and JHS if it makes sense for them as well.

> Add leveldb-based implementation for RMStateStore
> -------------------------------------------------
>                 Key: YARN-2765
>                 URL: https://issues.apache.org/jira/browse/YARN-2765
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-2765.patch, YARN-2765v2.patch
> It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb
would provide some benefits over the existing filesystem store such as better support for
atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem.

This message was sent by Atlassian JIRA

View raw message