hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-961) ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory load on TaskTracker(s)
Date Tue, 03 Nov 2009 01:01:03 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772802#action_12772802
] 

Matei Zaharia commented on MAPREDUCE-961:
-----------------------------------------

bq. I see. So we may need to count the previous continuously launched map tasks in canLauch
task and return a false if there are too many map tasks launched in a row. Is this correct?

Yes, either that or to change the scheduler to alternate between looking for a map and looking
for a reduce. Right now the logic in there is organized as:
{code}
for taskType in {MAP, REDUCE}:
  while true:
    if canLaunchTask(..., taskType):
     try to find a job with pending task
     if found a job:
       launch task
     else:
       break
{code}
It should become something like this:
{code}
while true:
  pick taskType to try next (if node has fewer maps than reduces, choose map; else choose
reduce)
  if canLaunchTask(..., taskType):
   try to find a job with pending task
   if found a job:
     launch task
   else:
     break
{code}

> ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory
load on TaskTracker(s)
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-961
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-961
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/fair-share
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: HIVE-961.patch, MAPREDUCE-961-v2.patch
>
>
> Design and develop a ResouceAwareLoadManager for the FairShare scheduler that dynamically
decides how many maps/reduces to run on a particular machine based on the CPU/Memory/diskIO/network
usage in that machine.  The amount of resources currently used on each task tracker is being
fed into the ResourceAwareLoadManager in real-time via an entity that is external to Hadoop.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message