hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-2108) Allow TaskScheduler manage number slots on TaskTrackers
Date Tue, 23 Nov 2010 17:54:29 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934937#action_12934937
] 

Arun C Murthy commented on MAPREDUCE-2108:
------------------------------------------

I like the direction.

I'll warn you that the current implementation of the JT has the notion of 'slot' burned in
very deeply into various parts of the codebase, not to mention the TT itself. You might need
to do a complete rewrite/refactor of the JT/TT to pull this off. My 2c. Good luck.

> Allow TaskScheduler manage number slots on TaskTrackers
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-2108
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2108
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/capacity-sched, contrib/fair-share
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>
>
> Currently the map slots and reduce slots are managed by TaskTracker configuration.
> To change the task tracker slots, we need to restart the TaskTrackers.
> Also, for a non-uniform cluster, we have to deploy different sets of configuration.
> Now JobTracker holds the CPU and memory status of TaskTrackers (MAPREDUCE-1218).
> So it makes sense to just let JobTracker.taskScheduler decided the number of slots on
each node.
> This way we can
> 1. Change the number of slots dynamically without restarting TaskTracker
> 2. Use different number of slots based on the resource of a TaskTracker
> To achieve this, we need to change the logic that we use totalMapSlots and totalReduceSlots
in JobTracker.
> I think they are used in WebUI and speculativeCap.
> We will need to make JobTracker calculate these numbers from TaskScheduler and TaskTrackerStatus.
> TaskScheduler and TaskTracker can both hold their maximum slots. We pick the smaller
one.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message