hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi OZAWA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1307) Rethink znode structure for RM HA
Date Thu, 24 Oct 2013 11:43:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13804088#comment-13804088

Tsuyoshi OZAWA commented on YARN-1307:

When the data size exceeds the size of a batch then a new znode is created.

Yes, we can take this apporach. However, let's marshal merits and demerits between batching
data and znode with encoded sequential numbers before doing this.

1. Batching data approach
* Merits:
** Number of znodes is reduced.
** No need to encode sequential numbers when storing
* Demerits:
** remove*() costs get much higher. We need to scan and deseliarize objects from upto 500KB
byte array for each znode. After doing this, we need to write updated data again.
** store*() costs gets a bit higher, because we need to read upto 500KB data and write it
back again.

2. Znodes with encoded sequential numbers approach:
* Merits:
** We can delete keys in O(1) when names of key are given.
** We can avoid reading data in znodes when storing/removing DelegationKeys or DelegatoinTokens.
* Demerits:
** If secret key for encoding is changed by user, stored secret keys gets invalid and need
to delete all data.

3. No change between them
*  Memory overhead, because no or subtle difference between data and znode.

IMO, znodes approach is better, because this approach can reduce ZooKeeper's server-side loads
compare to the other. One demerit is we need secret key to encode if we cannot encode sequence
numbers directly as I described.

If some points are wrong, please let me know.

> Rethink znode structure for RM HA
> ---------------------------------
>                 Key: YARN-1307
>                 URL: https://issues.apache.org/jira/browse/YARN-1307
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-1307.1.patch, YARN-1307.2.patch
> Rethink for znode structure for RM HA is proposed in some JIRAs(YARN-659, YARN-1222).
The motivation of this JIRA is quoted from Bikas' comment in YARN-1222:
> {quote}
> We should move to creating a node hierarchy for apps such that all znodes for an app
are stored under an app znode instead of the app root znode. This will help in removeApplication
and also in scaling better on ZK. The earlier code was written this way to ensure create/delete
happens under a root znode for fencing. But given that we have moved to multi-operations globally,
this isnt required anymore.
> {quote}

This message was sent by Atlassian JIRA

View raw message