hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Chen (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-961) ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory load on TaskTracker(s)
Date Mon, 09 Nov 2009 20:38:32 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Scott Chen updated MAPREDUCE-961:
---------------------------------

    Attachment: MAPREDUCE-961-v3.patch

Lots of things changed in this patch. 
1. Make MemoryCalculatorPlugin.java collect available memory on the TT.
2. The available memory information is added in TaskTrackerStatus.ResourceStatus and transmit
by heartbeat
3. MemBasedLoadManager launch the task if [Avalable memory - Max memory per task > Reserved
Memory]
    where Memory per Task is from existing configuration and reserved memory needs to be configured.

I have also fixed the problem that Matei pointed out to return super.canLaunchTask().

@Matei: About the other suggestion about alternatively launching different types of task,
I will open another JIRA and work on that. Because it is not part of MemBasedLoadManager.
Thanks.

> ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory
load on TaskTracker(s)
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-961
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-961
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/fair-share
>            Reporter: dhruba borthakur
>            Assignee: Scott Chen
>         Attachments: HIVE-961.patch, MAPREDUCE-961-v2.patch, MAPREDUCE-961-v3.patch
>
>
> Design and develop a ResouceAwareLoadManager for the FairShare scheduler that dynamically
decides how many maps/reduces to run on a particular machine based on the CPU/Memory/diskIO/network
usage in that machine.  The amount of resources currently used on each task tracker is being
fed into the ResourceAwareLoadManager in real-time via an entity that is external to Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message