hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
Date Sun, 31 Mar 2013 19:29:17 GMT

    [ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618446#comment-13618446
] 

Arun C Murthy commented on YARN-18:
-----------------------------------

[~vicaya] / [~djp] - Can you please describe what is happening here? I see a lot of factories
and then no changes to actually support node-group...

What do I need to read-up to understand what is going on here?

One thought - seems to me, (maybe longer term), we need to abstract the notion of topology
in the Scheduler without having to make changes every time we need to add another layer in
the topology... think of complex star-topologies etc.

Thoughts?

Another thing to consider is that, for now, it might be better to implement a new NodeGroup
Scheduler rather than change all existing Schedulers?
                
> Make locatlity in YARN's container assignment and task scheduling pluggable for other
deployment topology
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-18
>                 URL: https://issues.apache.org/jira/browse/YARN-18
>             Project: Hadoop YARN
>          Issue Type: New Feature
>    Affects Versions: 2.0.3-alpha
>            Reporter: Junping Du
>            Assignee: Junping Du
>              Labels: features
>         Attachments: HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, MAPREDUCE-4309.patch,
MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch,
MAPREDUCE-4309-v6.patch, MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch,
YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch,
YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch
>
>
> There are several classes in YARN’s container assignment and task scheduling algorithms
that relate to data locality which were updated to give preference to running a container
on other locality besides node-local and rack-local (like nodegroup-local). This propose to
make these data structure/algorithms pluggable, like: SchedulerNode, RMNodeImpl, etc. The
inner class ScheduledRequests was made a package level class to it would be easier to create
a subclass, ScheduledRequestsWithNodeGroup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message