Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 21 Oct 2016 04:31:59 +0000 (UTC)
From: "Arun Suresh (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12931339.1452828790000.48296.1477024319891@Atlassian.JIRA>
In-Reply-To: <JIRA.12931339.1452828790000@Atlassian.JIRA>
References: <JIRA.12931339.1452828790000@Atlassian.JIRA> <JIRA.12931339.1452828790896@arcas>
Subject: [jira] [Comment Edited] (YARN-4597) Add SCHEDULE to NM container
 lifecycle
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 21 Oct 2016 04:32:02 -0000


    [ https://issues.apache.org/jira/browse/YARN-4597?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D15593=
945#comment-15593945 ]=20

Arun Suresh edited comment on YARN-4597 at 10/21/16 4:31 AM:
-------------------------------------------------------------

[~jianhe], thanks again for taking a look.

bq. I think there might be some behavior change or bug for scheduling guara=
nteed containers when the oppotunistic-queue is enabled. Previously, when l=
aunching container, NM will not check for current vmem usage, and cpu usage=
. It assumes what RM allocated can be launched.
Now, NM will check these limits and won't launch the container if hits the =
limit.
Yup, we do a *hasResources* check only at the start of a container and when=
 a container is killed. We assumed that resources requested by a container =
is constant, essentially we considered only actual *allocated* resources wh=
ich we assume will not varying during the lifetime of the container... whic=
h implies, there is no point in checking this at any other time other than =
start and kill of containers.
But like you stated, if we consider container resource *utilization*, based=
 on the work [~kasha] is doing in YARN-1011, then yes, we should have a tim=
er thread that periodically checks the vmem and cpu usage and starts (and k=
ills) containers based on that.

bq. the ResourceUtilizationManager looks like only incorporated some utilit=
y methods, not sure how we will make this pluggable later.
Following on my point above, the idea was to have a {{ResourceUtilizationMa=
nager}} that can provide a different value of {{getCurrentUtilization}}, {{=
addResource}} and {{subtractResource}} which is used by the ContainerSchedu=
ler to calculate the resources to free up. For instance, the current defaul=
t one only takes into account actual resource *allocated* to containers... =
 for YARN-1011, we might replace that with the resource *utilized* by runni=
ng containers, and provide a different value for {{getCurrentUtilization}}.=
 The timer thread I mentioned in the previous point, which can be apart of =
this new ResourceUtilizationManager, can send events to the scheduler to re=
-process queued containers when utilization has changed.

bq. The logic to select opportunisitic container: we may kill more opportun=
istic containers than required. e.g...
Good catch, in the {{resourcesToFreeUp}}, I needed to decrement any already=
-marked-for-kill opportunistic container. It was there earlier, Had removed=
 it when I was testing something, but forgot to put it back :)

bq. we don't need to synchronize on the currentUtilization object? I don't =
see any other place it's synchronized
Yup, It isnt required. Varun did point out the same.. I thought I had fixed=
 it, think I might have missed 'git add'ing the change

w.r.t Adding the new transitions, I was seeing some error messages in some =
testcases. Will rerun and see if they are required=E2=80=A6 but in anycase,=
 having them there should be harmless right?
=20
The rest of your comments makes sense.. will address them shortly.


was (Author: asuresh):
[~jianhe], thanks again for taking a look.

bq. I think there might be some behavior change or bug for scheduling guara=
nteed containers when the oppotunistic-queue is enabled.
Previously, when launching container, NM will not check for current vmem us=
age, and cpu usage. It assumes what RM allocated can be launched.
Now, NM will check these limits and won't launch the container if hits the =
limit.
Yup, we do a *hasResources* check only at the start of a container and when=
 a container is killed. We assumed that resources requested by a container =
is constant, essentially we considered only actual *allocated* resources wh=
ich we assume will not varying during the lifetime of the container... whic=
h implies, there is no point in checking this at any other time other than =
start and kill of containers.
But like you stated, if we consider container resource *utilization*, based=
 on the work [~kasha] is doing in YARN-1011, then yes, we should have a tim=
er thread that periodically checks the vmem and cpu usage and starts (and k=
ills) containers based on that.

bq. the ResourceUtilizationManager looks like only incorporated some utilit=
y methods, not sure how we will make this pluggable later.
Following on my point above, the idea was to have a {{ResourceUtilizationMa=
nager}} that can provide a different value of {{getCurrentUtilization}}, {{=
addResource}} and {{subtractResource}} which is used by the ContainerSchedu=
ler to calculate the resources to free up. For instance, the current defaul=
t one only takes into account actual resource *allocated* to containers... =
 for YARN-1011, we might replace that with the resource *utilized* by runni=
ng containers, and provide a different value for {{getCurrentUtilization}}.=
 The timer thread I mentioned in the previous point, which can be apart of =
this new ResourceUtilizationManager, can send events to the scheduler to re=
-process queued containers when utilization has changed.

bq. The logic to select opportunisitic container: we may kill more opportun=
istic containers than required. e.g...
Good catch, in the {{resourcesToFreeUp}}, I needed to decrement any already=
-marked-for-kill opportunistic container. It was there earlier, Had removed=
 it when I was testing something, but forgot to put it back :)

bq. we don't need to synchronize on the currentUtilization object? I don't =
see any other place it's synchronized
Yup, It isnt required. Varun did point out the same.. I thought I had fixed=
 it, think I might have missed 'git add'ing the change

w.r.t Adding the new transitions, I was seeing some error messages in some =
testcases. Will rerun and see if they are required=E2=80=A6 but in anycase,=
 having them there should be harmless right?
=20
The rest of your comments makes sense.. will address them shortly.


> Add SCHEDULE to NM container lifecycle
> --------------------------------------
>
>                 Key: YARN-4597
>                 URL: https://issues.apache.org/jira/browse/YARN-4597
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: Chris Douglas
>            Assignee: Arun Suresh
>         Attachments: YARN-4597.001.patch, YARN-4597.002.patch, YARN-4597.=
003.patch
>
>
> Currently, the NM immediately launches containers after resource localiza=
tion. Several features could be more cleanly implemented if the NM included=
 a separate stage for reserving resources.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org