hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
Date Thu, 29 Nov 2012 15:23:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506529#comment-13506529

Junping Du commented on YARN-18:

Hi Arun, Thanks for your review on this patch. This patch works with YARN-19 to make YARN
can aware "node-group" level (the layer between host and rack which is mostly for cloud/virtualization
use case) data locality. In previous work on Common and HDFS part for "node-group", Nicholas
suggest me to divide changes into "pluggable" and "plug-in" patches which make previous 2
layers (host/rack) more pluggable and support new layer in plug-in which can be enabled with
specific configuration. 
So for this patch, it tries to make locality related interface/class can be extensible with
node-group layer like: ScheduledRequests, RMNodeImpl, SchedulerNode, etc. Also, it make some
methods can be overridden in sub-class like scheduler's assignContainer() method. In sub class,
we can make it allocate container for nodegroup-local tasks before allocating rack-local tasks.
Also, this patch add more unit tests. Please let me know if anything here make you confusing.
> Make locatlity in YARN's container assignment and task scheduling pluggable for other
deployment topology
> ---------------------------------------------------------------------------------------------------------
>                 Key: YARN-18
>                 URL: https://issues.apache.org/jira/browse/YARN-18
>             Project: Hadoop YARN
>          Issue Type: New Feature
>    Affects Versions: 2.0.3-alpha
>            Reporter: Junping Du
>            Assignee: Junping Du
>              Labels: features
>         Attachments: HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, MAPREDUCE-4309.patch,
MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch,
MAPREDUCE-4309-v6.patch, MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch
> There are several classes in YARN’s container assignment and task scheduling algorithms
that relate to data locality which were updated to give preference to running a container
on other locality besides node-local and rack-local (like nodegroup-local). This propose to
make these data structure/algorithms pluggable, like: SchedulerNode, RMNodeImpl, etc. The
inner class ScheduledRequests was made a package level class to it would be easier to create
a subclass, ScheduledRequestsWithNodeGroup.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message