hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantinos Karanasos (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2883) Queuing of container requests in the NM
Date Wed, 23 Mar 2016 18:07:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208893#comment-15208893

Konstantinos Karanasos commented on YARN-2883:

Thanks for the feedback, [~chris.douglas] and [~kasha]!

I am in the process of addressing Chris' comments -- will upload a new patch soon.

Regarding Karthik's comments:
bq. Any reason we use a map instead of a queue to store the queued containers?
I am using a Map only to track the allocated containers; for the queued containers, I am using
a queue, as you suggest.

bq. I like that QueuingContainerManagerImpl extends ContainerManagerImpl - while we harden
the queuing side of things, it will help keep the code clean. In the longer run, we might
want to default to Queuing implementation and play with the queue length, but we can cross
that bridge when we get there.
Agreed, that was exactly our intention too.

bq. IIUC, the intent is to use queueing for all opportunistic containers. The ContainerManagerImpl
implementation seems to depend on whether queuing is enabled - wouldn't that affect all containers
and not just opportunistic containers?
In most cases (including distributed scheduling and resource over-commitment), queues will
indeed only be used for opportunistic containers.
However, as long as queuing is enabled, guaranteed containers might need to be queued momentarily
until the opportunistic containers that block their execution get killed.
That's the reason you see guaranteed containers going through the same code-path too. But
again, this will not break any semantics of the guaranteed containers.

bq. The patch has the author's name left against a TODO. Also, we don't want to leave orphaned
TODOs - let us go ahead and file a JIRA
True, I will make sure I remove any TODOs and author names.

bq. The ResourceUtilization changes are not strictly related to this patch, do they?
This is correct. I put them in this JIRA because they are just a couple of methods. Do you
think I should create a separate JIRA for this?

bq. TestQueuingContainerMgr: We typically don't wrap imports at 80 chars.
Yep, will fix that.

> Queuing of container requests in the NM
> ---------------------------------------
>                 Key: YARN-2883
>                 URL: https://issues.apache.org/jira/browse/YARN-2883
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Konstantinos Karanasos
>            Assignee: Konstantinos Karanasos
>         Attachments: YARN-2883-trunk.004.patch, YARN-2883-yarn-2877.001.patch, YARN-2883-yarn-2877.002.patch,
YARN-2883-yarn-2877.003.patch, YARN-2883-yarn-2877.004.patch
> We propose to add a queue in each NM, where queueable container requests can be held.
> Based on the available resources in the node and the containers in the queue, the NM
will decide when to allow the execution of a queued container.
> In order to ensure the instantaneous start of a guaranteed-start container, the NM may
decide to pre-empt/kill running queueable containers.

This message was sent by Atlassian JIRA

View raw message