hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Chen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-961) ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory load on TaskTracker(s)
Date Mon, 12 Oct 2009 19:11:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12764791#action_12764791
] 

Scott Chen commented on MAPREDUCE-961:
--------------------------------------

1. This patch implements class org.apache.hadoop.mapred.MemBasedLoadManager which looks at
the memory usage of the TaskTracker and the Job of the task to determine whether to load this
task.
2. This patch also contains a resource utilization monitoring daemon which run on each of
the TaskTrackers and a information collecting daemon called Collector which collects and aggregates
the resource utilization information submitted from each monitoring daemons. 

Note that this is somewhat similar to org.apache.hadoop.mapred.TaskMemoryManagerThread but
different. Because we are collecting information based both on TaskTracker and Jobs. TaskMemoryManagerThread
does not know the memory usage based on Hadoop job. Also, we are reporting the information
back to a master called Collector and resolve the memory issue in the jobtracker level where
TaskMemoryManagerThread resolves this issue in the tasktracker level.

> ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory
load on TaskTracker(s)
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-961
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-961
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/fair-share
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: HIVE-961.patch
>
>
> Design and develop a ResouceAwareLoadManager for the FairShare scheduler that dynamically
decides how many maps/reduces to run on a particular machine based on the CPU/Memory/diskIO/network
usage in that machine.  The amount of resources currently used on each task tracker is being
fed into the ResourceAwareLoadManager in real-time via an entity that is external to Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message