hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-2765) Add leveldb-based implementation for RMStateStore
Date Wed, 29 Oct 2014 15:50:34 GMT

     [ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Lowe updated YARN-2765:
-----------------------------
    Attachment: YARN-2765v2.patch

Thanks for the review, Tsuyoshi!

bq. How about adding helper methods like getKeyPrefix/getNodePath for getting key prefix and
node path?

Sure, added some helper methods to compute leveldb keys for various things.

bq. I found that the patch includes lots hard-coded "/". I think it's better to have private
field SEPARATOR = "/". 

IMHO this makes the code less readable, similar to a code style like {{final int ONE = 1}}.
 But I don't care too strongly about it and changed all occurrences to SEPARATOR.

For Zhijie's comments:

bq. One drawback I can think of is that while LeveldbRMStateStore is lightweight for single
RM restarting, multiple RMs of HA are not able to share this single-host database.

This should work if the leveldb database is on a network store like a filer.  Leveldb uses
locks to prevent multiple processes from trying to access the database simultaneously, so
there's a little bit of help for the fencing scenarios.  However the fencing script actions
would have to do some extra work to force a poorly-behaving resourcemanager to let go of the
locks so a standby RM can open the store and become active.

bq. Did you have a chance to think of an enhanced k/v db: rocksdb?

I briefly considered using rocksdb for this but decided against it for a couple of reasons:

* leveldb is already used by the timeline server and nodemanager, and I would rather avoid
adding yet another new dependency for this
* leveldb supports win32/win64, but it doesn't appear that the standard rocksdbjni distribution
has support for Windows.

> Add leveldb-based implementation for RMStateStore
> -------------------------------------------------
>
>                 Key: YARN-2765
>                 URL: https://issues.apache.org/jira/browse/YARN-2765
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-2765.patch, YARN-2765v2.patch
>
>
> It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb
would provide some benefits over the existing filesystem store such as better support for
atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message