Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Mon, 4 May 2015 01:14:07 +0000 (UTC)
From: "Vinod Kumar Vavilapalli (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12761770.1418658644000.66593.1430702047513@Atlassian.JIRA>
In-Reply-To: <JIRA.12761770.1418658644000@Atlassian.JIRA>
References: <JIRA.12761770.1418658644000@Atlassian.JIRA>
 <JIRA.12761770.1418658644928@arcas>
Subject: [jira] [Commented] (YARN-2962) ZKRMStateStore: Limit the number of
 znodes under a znode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526130#comment-14526130 ] 

Vinod Kumar Vavilapalli commented on YARN-2962:
-----------------------------------------------

bq. it will indeed be backwards incompatible. We should get this into trunk. To include this in branch-2, we should probably guard this with a config, and users will have to format the store to start using this. Otherwise, an RM restart will fail.

Apologies for coming in late. I don't think we should give up on compatibility that easily. You can easily provide compat by doing one of the following

 - Forcing the new structure into its own parent directly. For e.g. instead of directly under RM_APP_ROOT, put it under something like RM_APP_ROOT/hierarchies, that way RM can read the old and new styles. /cc [~jianhe].
 - If not that, we can simplify it without special hierarchies by having RM recursively read nodes all the time. As we already specifically look for "application_" prefix to read app-date, having application_1234_0456 right next to a 00/ directory is simply going to work without much complexity.

That way, we should also have a more reasonable default instead of the current no-split default config value.

> ZKRMStateStore: Limit the number of znodes under a znode
> --------------------------------------------------------
>
>                 Key: YARN-2962
>                 URL: https://issues.apache.org/jira/browse/YARN-2962
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 2.6.0
>            Reporter: Karthik Kambatla
>            Assignee: Varun Saxena
>            Priority: Critical
>         Attachments: YARN-2962.01.patch, YARN-2962.2.patch, YARN-2962.3.patch
>
>
> We ran into this issue where we were hitting the default ZK server message size configs, primarily because the message had too many znodes even though they individually they were all small.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)