hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-961) ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory load on TaskTracker(s)
Date Tue, 03 Nov 2009 01:02:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772805#action_12772805
] 

Matei Zaharia commented on MAPREDUCE-961:
-----------------------------------------

Actually I have a minor bug in the second code snippet... instead of picking one task type
and breaking, you should sort the task types by whoever has the least number of active tasks
on the node, and try both task types. For example, if there are fewer maps on the node than
reduces, first look for a map, and then for a reduce. Only break if neither task type has
tasks to launch.

> ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory
load on TaskTracker(s)
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-961
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-961
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/fair-share
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: HIVE-961.patch, MAPREDUCE-961-v2.patch
>
>
> Design and develop a ResouceAwareLoadManager for the FairShare scheduler that dynamically
decides how many maps/reduces to run on a particular machine based on the CPU/Memory/diskIO/network
usage in that machine.  The amount of resources currently used on each task tracker is being
fed into the ResourceAwareLoadManager in real-time via an entity that is external to Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message