hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-2765) Add leveldb-based implementation for RMStateStore
Date Mon, 01 Dec 2014 21:16:13 GMT

     [ https://issues.apache.org/jira/browse/YARN-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Lowe updated YARN-2765:
-----------------------------
    Attachment: YARN-2765v3.patch

Thanks for the review, Jian!

bq. Patch needs updated on top of YARN-2404.
Done.

{quote}
DB_SCHEMA_VERSION_KEY -> VERSION_NODE
EPOCH_KEY -> EPOCH_NODE
{quote}
Done.

bq. RM_DT_SEQUENCE_NUMBER_KEY: use DELEGATION_TOKEN_SEQUENCE_NUMBER_PREFIX for “RMDTSequentialNumber”

I did not make this change.  Unlike the filesystem state store we are not using the key to
encode the data but rather just as a static key that is atomically updated with new values.
 Therefore it's not appropriate to use a key that ends in a prefix, IMHO.  This store is closer
to what the ZK store is doing which also uses a separate key string for the sequence number.

bq. this check is always true and so unnecessary, as attemptState is earlier initialized with
attemptId

Left this check since after the refactoring for YARN-2404 the check will verify the attempt
ID parsed from the key matches the attempt ID parsed from the value.

{quote}
    Some naming suggestions:
    loadRMAppAttemptState-> createAttemptState
    loadRMAppState(String appIdStr, byte[] data) -> createApplicationState
    String itemName = key.substring(keyPrefix.length()); rename itemName to attemptId, and
keyPrefix to attemptNodePrefix
{quote}
Done.

bq. Question: does the levelDB client API handle retry ?
No, there are no retries in leveldb itself AFAIK.  The retries are handled by the OS/driver/device
layers, as we are hitting the OS filesystem rather than some network layer like HDFS or ZK
does.  Retries would be highly unlikely to be successful at the leveldb or app level.

> Add leveldb-based implementation for RMStateStore
> -------------------------------------------------
>
>                 Key: YARN-2765
>                 URL: https://issues.apache.org/jira/browse/YARN-2765
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-2765.patch, YARN-2765v2.patch, YARN-2765v3.patch
>
>
> It would be nice to have a leveldb option to the resourcemanager recovery store. Leveldb
would provide some benefits over the existing filesystem store such as better support for
atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message