Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Tue, 11 Nov 2014 21:01:33 +0000 (UTC)
From: "Craig Welch (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12754574.1415739675000.472697.1415739693619@Atlassian.JIRA>
In-Reply-To: <JIRA.12754574.1415739675000@Atlassian.JIRA>
References: <JIRA.12754574.1415739675000@Atlassian.JIRA>
 <JIRA.12754574.1415739675063@arcas>
Subject: [jira] [Created] (YARN-2848) (FICA) Applications should maintain an
 application specific 'cluster' resource to calculate headroom and userlimit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Craig Welch created YARN-2848:
---------------------------------

             Summary: (FICA) Applications should maintain an application sp=
ecific 'cluster' resource to calculate headroom and userlimit
                 Key: YARN-2848
                 URL: https://issues.apache.org/jira/browse/YARN-2848
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: capacityscheduler
            Reporter: Craig Welch
            Assignee: Craig Welch


Likely solutions to [YARN-1680] (properly handling node and rack blacklisti=
ng with cluster level node additions and removals) will entail managing an =
application-level "slice" of the cluster resource available to the applicat=
ion for use in accurately calculating the application headroom and user lim=
it.  There is an assumption that events which impact this resource will cha=
nge less frequently than the need to calculate headroom, userlimit, etc (wh=
ich is a valid assumption given that occurs per-allocation heartbeat).  Giv=
en that, the application should (with assistance from cluster-level code...=
) detect changes to the composition of the cluster (node addition, removal)=
 and when those have occurred, calculate a application specific cluster res=
ource by comparing cluster nodes to it's own blacklist (both rack and indiv=
idual node).  I think it makes sense to include nodelabel considerations in=
to this calculation as it will be efficient to do both at the same time and=
 the single resource value reflecting both constraints could then be used f=
or efficient frequent headroom and userlimit calculations while remaining h=
ighly accurate.  The application would need to be made aware of nodelabel c=
hanges it is interested in (the application or removal of labels of interes=
t to the application to/from nodes).  For this purpose, the application sub=
missions's nodelabel expression would be used to determine the nodelabel im=
pact on the resource used to calculate userlimit and headroom (Cases where =
application elected to request resources not using the application level la=
bel expression are out of scope for this - but for the common usecase of an=
 application which uses a particular expression throughout, userlimit and h=
eadroom would be accurate) This could also provide an overall mechanism for=
 handling application-specific resource constraints which might be added in=
 the future.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)