hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1307) Rethink znode structure for RM HA
Date Sun, 15 Dec 2013 17:25:07 GMT

    [ https://issues.apache.org/jira/browse/YARN-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13848641#comment-13848641

Bikas Saha commented on YARN-1307:

its going to help scale zk store now that we are storing apps for a longer period. with 10K
apps there will be 20K-30K znodes created under the root znode. I dont know if anyone has
tried creating so many znodes under a single parent. IIRC, the number of znodes needed to
store (and thus retrieve) the data should be the same as earlier (1 per app and app-attempt)
Thus we should not be doing many more zk operations compared to the current structure. [~ozawa]?

> Rethink znode structure for RM HA
> ---------------------------------
>                 Key: YARN-1307
>                 URL: https://issues.apache.org/jira/browse/YARN-1307
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-1307.1.patch, YARN-1307.10.patch, YARN-1307.11.patch, YARN-1307.2.patch,
YARN-1307.3.patch, YARN-1307.4-2.patch, YARN-1307.4-3.patch, YARN-1307.4.patch, YARN-1307.5.patch,
YARN-1307.6.patch, YARN-1307.7.patch, YARN-1307.8.patch, YARN-1307.9.patch, YARN-1307.9.patch
> Rethink for znode structure for RM HA is proposed in some JIRAs(YARN-659, YARN-1222).
The motivation of this JIRA is quoted from Bikas' comment in YARN-1222:
> {quote}
> We should move to creating a node hierarchy for apps such that all znodes for an app
are stored under an app znode instead of the app root znode. This will help in removeApplication
and also in scaling better on ZK. The earlier code was written this way to ensure create/delete
happens under a root znode for fencing. But given that we have moved to multi-operations globally,
this isnt required anymore.
> {quote}

This message was sent by Atlassian JIRA

View raw message