hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abin Shahab (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-19) 4-layer topology (with NodeGroup layer) implementation of Container Assignment and Task Scheduling (for YARN)
Date Wed, 03 Jun 2015 20:59:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571632#comment-14571632

Abin Shahab commented on YARN-19:

Hi [~jdu] Can this be merged in 2.8? We at Altiscale are encountering issues with split NM
and DN. If YARN does not know how to schedule a container based on topology, locality suffers.
[~raviprakash] and [~aw], what do you guys think?

> 4-layer topology (with NodeGroup layer) implementation of Container Assignment and Task
Scheduling (for YARN)
> -------------------------------------------------------------------------------------------------------------
>                 Key: YARN-19
>                 URL: https://issues.apache.org/jira/browse/YARN-19
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Junping Du
>            Assignee: Junping Du
>         Attachments: HADOOP-8475-ContainerAssignmentTaskScheduling-withNodeGroup.patch,
MAPREDUCE-4310-v1.patch, MAPREDUCE-4310.patch, YARN-19-v2.patch, YARN-19-v3-alpha.patch, YARN-19-v4.patch,
> There are several classes in YARN’s container assignment and task scheduling algorithms
that related to data locality which were updated to give preference to running a container
on the same nodegroup. This section summarized the changes in the patch that provides a new
implementation to support a four-layer hierarchy.
> When the ApplicationMaster makes a resource allocation request to the scheduler of ResourceManager,
it will add the node group to the list of attributes in the ResourceRequest. The parameters
of the resource request will change from <priority, (host, rack, *), memory, #containers>
to <priority, (host, nodegroup, rack, *), memory, #containers>.
> After receiving the ResoureRequest the RM scheduler will assign containers for requests
in the sequence of data-local, nodegroup-local, rack-local and off-switch.Then, ApplicationMaster
schedules tasks on allocated containers in sequence of data- local, nodegroup-local, rack-local
and off-switch.
> In terms of code changes made to YARN task scheduling, we updated the class ContainerRequestEvent
so that applications can requests for containers can include anodegroup. In RM schedulers,
FifoScheduler and CapacityScheduler were updated. For the FifoScheduler, the changes were
in the method assignContainers. For the Capacity Scheduler the method assignContainersOnNode
in the class of LeafQueue was updated. In both changes a new method, assignNodeGroupLocalContainers()
was added in between the assignment data-local and rack-local.

This message was sent by Atlassian JIRA

View raw message