hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
Date Thu, 30 Jul 2015 18:09:04 GMT

     [ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wangda Tan updated YARN-3983:
-----------------------------
    Attachment: YARN-3983.2.patch

Thanks for comments, [~jianhe].

Addressed all comments except two:
1. move the assingContainersOnNode into internalApplyAllocation
assignContainersOnNode is not do the allocate, it is actually "checkIfContainerCanBeAllocatedOnNode".
I prefer to keep it in preAllocation.

2. getCSAssignmentFromAllocateResult can be part of doAllocation.
It may be better to be kept at FiCaSchedulerApp: ContainerAllocation contains some operations
such as SKIP_PRIORITY will be only used by application, CSAssignment doesn't have filed like
that. (There's a "skipped" in CSAssignment, but FiCaSchedulerApp doesn't know it is skipped
for priority/application/queue.

Attached ver.2 patch.

> Make CapacityScheduler to easier extend application allocation logic
> --------------------------------------------------------------------
>
>                 Key: YARN-3983
>                 URL: https://issues.apache.org/jira/browse/YARN-3983
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-3983.1.patch, YARN-3983.2.patch
>
>
> While working on YARN-1651 (resource allocation for increasing container), I found it
is very hard to extend existing CapacityScheduler resource allocation logic to support different
types of resource allocation.
> For example, there's a lot of differences between increasing a container and allocating
a container:
> - Increasing a container doesn't need to check locality delay.
> - Increasing a container doesn't need to build/modify a resource request tree (ANY->RACK/HOST).
> - Increasing a container doesn't need to check allocation/reservation starvation (see
{{shouldAllocOrReserveNewContainer}}).
> - After increasing a container is approved by scheduler, it need to update an existing
container token instead of creating new container.
> And there're lots of similarities when allocating different types of resources.
> - User-limit/queue-limit will be enforced for both of them.
> - Both of them needs resource reservation logic. (Maybe continuous reservation looking
is needed for both of them).
> The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation
logic to support different types of resource allocation, make common code reusable, and also
better code organization.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message