hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-1197) Support changing resources of an allocated container
Date Mon, 25 Nov 2013 04:40:40 GMT

     [ https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wangda Tan updated YARN-1197:
-----------------------------

    Attachment: yarn-server-resourcemanager.patch.ver.1
                yarn-server-nodemanager.patch.ver.1
                yarn-server-common.patch.ver.1
                yarn-pb-impl.patch.ver.1
                yarn-api-protocol.patch.ver.1
                tools-project.patch.ver.1
                mapreduce-project.patch.ver.1

I just finished container resource increase support including PB/API changes, make capacity
scheduler support increasing and NM can support change monitoring size of a running container.

*I splitted it to several patches for easier review,*
* API/pb file changes in hadoop-yarn-api
* PB implementations in hadoop-yarn-common
* yarn-server-common changes 
* yarn-server-resourcemanager changes include capacity scheduler and AMS master, etc. changes
 
* yarn-server-nodemanager changes include ContainerManagerImpl and ContainersMonitor changes
* other related project changes according to updated APIs (map-reduce/tools)
Aboves a preview patches, still very rough, [~bikassaha], [~sandyr], [~tucu00] , [~vinodkv]
could you please do some review on them, I'm eager for your ideas!

*And some short notes for current implementations on RM/NM not covered in design doc,*
*1) Implementation in capacity scheduler for increasing a container size*
It's very close to allocate a new container, some details,
* Increase request can be only valid when asked size larger than existed resource, and container
state is either RUNNING or ACQUIRED
* The entry point of increase request allocation is still in CapacityScheduler:nodeUpdate()
* When increase request cannot be allocated, it will also be reserved. Each node can only
reserve at most one request (increase request or new container request). I created a new method
isReserved() in FiCaSchedulerNode to make scheduler/queue identify if a node is reserved
* The major logic for increase request allocation is also placed in LeafQueue:assignContainers,
increase requests will be proceeded before new container request.
* Queue(leaf/parent) capacity and user capacity checking will also be done before reserve
or allocate a increase request
* Queue(leaf/parent) used resource will also be deduct when increase request reserved
* Users may submit increase request several times on a same container with different size.
** If asked size is equal to previous asked size, it will be ignored 
** If asked size is smaller or equal to existed size, this will cancel increase request on
this container
** If asked size is different of previous asked size, and greater than existing size, it will
replace previous ask and cancel previous reservations (if existed).

*2) Implementation in node manager for increasing a container size*
* It will do a similar check logic (like token verifications, etc.) like start container
* Increase logic will only valid when ContainerState(The internal ContainerState: org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerState)
is RUNNING, to avoid a racing condition.
* ContainersMonitorImpl will put change requests to containersToBeChanged when received CHANGE_MONITORING_CONTAINER
event. And it will be proceeded in MonitoringThread:run()

> Support changing resources of an allocated container
> ----------------------------------------------------
>
>                 Key: YARN-1197
>                 URL: https://issues.apache.org/jira/browse/YARN-1197
>             Project: Hadoop YARN
>          Issue Type: Task
>          Components: api, nodemanager, resourcemanager
>    Affects Versions: 2.1.0-beta
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: mapreduce-project.patch.ver.1, tools-project.patch.ver.1, yarn-1197-v2.pdf,
yarn-1197-v3.pdf, yarn-1197-v4.pdf, yarn-1197.pdf, yarn-api-protocol.patch.ver.1, yarn-pb-impl.patch.ver.1,
yarn-server-common.patch.ver.1, yarn-server-nodemanager.patch.ver.1, yarn-server-resourcemanager.patch.ver.1
>
>
> Currently, YARN cannot support merge several containers in one node to a big container,
which can make us incrementally ask resources, merge them to a bigger one, and launch our
processes. The user scenario is described in the comments.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message