Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Mon, 21 Mar 2016 06:19:25 +0000 (UTC)
From: "Wangda Tan (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12951947.1458541162000.1307.1458541165651@Atlassian.JIRA>
In-Reply-To: <JIRA.12951947.1458541162000@Atlassian.JIRA>
References: <JIRA.12951947.1458541162000@Atlassian.JIRA>
 <JIRA.12951947.1458541162401@arcas>
Subject: [jira] [Created] (YARN-4844) Upgrade fields of
 o.a.h.y.api.records.Resource from int32 to int64
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Wangda Tan created YARN-4844:
--------------------------------

             Summary: Upgrade fields of o.a.h.y.api.records.Resource from int32 to int64
                 Key: YARN-4844
                 URL: https://issues.apache.org/jira/browse/YARN-4844
             Project: Hadoop YARN
          Issue Type: Sub-task
          Components: api
            Reporter: Wangda Tan
            Priority: Critical


We use int32 for memory now, if a cluster has 10k nodes, each node has 210G memory, we will get a negative total cluster memory.

And another case that easier overflows int32 is: we added all pending resources of running apps to cluster's total pending resources. If a problematic app requires too much resources (let's say 1M+ containers, each of them has 3G containers), int32 will be not enough.

Even if we can cap each app's pending request, we cannot handle the case that there're many running apps, each of them has capped but still significant numbers of pending resources.

So we may possibly need to upgrade int32 memory field (could include v-cores as well) to int64 to avoid integer overflow. 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)