hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2962) ZKRMStateStore: Limit the number of znodes under a znode
Date Mon, 04 May 2015 01:14:07 GMT

    [ https://issues.apache.org/jira/browse/YARN-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526130#comment-14526130

Vinod Kumar Vavilapalli commented on YARN-2962:

bq. it will indeed be backwards incompatible. We should get this into trunk. To include this
in branch-2, we should probably guard this with a config, and users will have to format the
store to start using this. Otherwise, an RM restart will fail.

Apologies for coming in late. I don't think we should give up on compatibility that easily.
You can easily provide compat by doing one of the following

 - Forcing the new structure into its own parent directly. For e.g. instead of directly under
RM_APP_ROOT, put it under something like RM_APP_ROOT/hierarchies, that way RM can read the
old and new styles. /cc [~jianhe].
 - If not that, we can simplify it without special hierarchies by having RM recursively read
nodes all the time. As we already specifically look for "application_" prefix to read app-date,
having application_1234_0456 right next to a 00/ directory is simply going to work without
much complexity.

That way, we should also have a more reasonable default instead of the current no-split default
config value.

> ZKRMStateStore: Limit the number of znodes under a znode
> --------------------------------------------------------
>                 Key: YARN-2962
>                 URL: https://issues.apache.org/jira/browse/YARN-2962
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 2.6.0
>            Reporter: Karthik Kambatla
>            Assignee: Varun Saxena
>            Priority: Critical
>         Attachments: YARN-2962.01.patch, YARN-2962.2.patch, YARN-2962.3.patch
> We ran into this issue where we were hitting the default ZK server message size configs,
primarily because the message had too many znodes even though they individually they were
all small.

This message was sent by Atlassian JIRA

View raw message