hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Chen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-961) ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory load on TaskTracker(s)
Date Fri, 13 Nov 2009 20:40:39 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777644#action_12777644
] 

Scott Chen commented on MAPREDUCE-961:
--------------------------------------

Thanks for the comment, Arun. I have changed the patch a lot following the suggestion froms
Matei and Vinod. The last patch is total different from the first one. I am sorry about the
confusion.

The following is the design
1. We obtain the available memory on the TT using MemoryCalculatorPlugin. Originally this
class calculates only total memory only, we add a slight change so that it also computes the
available memory.
2. The information is reported with TaskTrackerStatus.ResourceStatus back to JT.
3. In MemBasedLoadManager, we look at the available memory on TT, the maximum memory per task
(from jobConf) and a configured reserved memory on TT. If (available memory - task memory
> reserved memory), we return true which allows scheduler to lauch the task.

The initial idea also includes using the memory usage of a job collecting in the cluster.
Right now we only use the value obtained in jobConf. After MAPREDUCE-220 is done, we can use
the task memory estimated by the previous tasks.

> ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory
load on TaskTracker(s)
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-961
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-961
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/fair-share
>    Affects Versions: 0.22.0
>            Reporter: dhruba borthakur
>            Assignee: Scott Chen
>             Fix For: 0.22.0
>
>         Attachments: HIVE-961.patch, MAPREDUCE-961-v2.patch, MAPREDUCE-961-v3.patch,
MAPREDUCE-961-v4.patch
>
>
> Design and develop a ResouceAwareLoadManager for the FairShare scheduler that dynamically
decides how many maps/reduces to run on a particular machine based on the CPU/Memory/diskIO/network
usage in that machine.  The amount of resources currently used on each task tracker is being
fed into the ResourceAwareLoadManager in real-time via an entity that is external to Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message