hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1307) Rethink znode structure for RM HA
Date Wed, 23 Oct 2013 19:01:49 GMT

    [ https://issues.apache.org/jira/browse/YARN-1307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803148#comment-13803148
] 

Bikas Saha commented on YARN-1307:
----------------------------------

It is fine to store a batch of delegation token in the same znode. In fact that may be better
than creating a huge number of znode. A setData operation is easy to do on a znode. So the
stateStore can keep the current (upto 500kb) batch in memory. When a new delegation token
arrives then the store can add it to the batch and overwrite the znode data with the new batch.
Keep doing it till the batch size is ~500kb. When the data size exceeds the size of a batch
then a new znode is created.

There is no need to encode the sequence number in the name. It can be made part of the znode
data. The sequence number znode stores the sequence number of the tokens. We encoded the sequence
number in the name for HDFS so that we dont have to go to the data nodes to write the data.
this is not necessary in ZK since it supports a setData operation that overwrites the previous
data. We can simply name the znode something like RMDelegationTokenSequenceNumber.



> Rethink znode structure for RM HA
> ---------------------------------
>
>                 Key: YARN-1307
>                 URL: https://issues.apache.org/jira/browse/YARN-1307
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-1307.1.patch, YARN-1307.2.patch
>
>
> Rethink for znode structure for RM HA is proposed in some JIRAs(YARN-659, YARN-1222).
The motivation of this JIRA is quoted from Bikas' comment in YARN-1222:
> {quote}
> We should move to creating a node hierarchy for apps such that all znodes for an app
are stored under an app znode instead of the app root znode. This will help in removeApplication
and also in scaling better on ZK. The earlier code was written this way to ensure create/delete
happens under a root znode for fencing. But given that we have moved to multi-operations globally,
this isnt required anymore.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message