hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-961) ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory load on TaskTracker(s)
Date Wed, 14 Oct 2009 06:31:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12765422#action_12765422

Vinod K V commented on MAPREDUCE-961:

Hastily looked at the patch. Some comments overall.
 - We *already have a complete reporting framework in the form of HeartBeats* and don't think
we need new setup of daemons yet.
   -- _InterTrackerProtocol_ can substitute _CollectorProtocol_
    -- Reporting can be done via _TaskTrackerStatus.ResourceStatus_ at the TaskTracker via
    -- Aggregate collection can be via the same _TaskTrackerStatus.ResourceStatus_ at the
JobTracker via HeartBeat
    -- Getters for the utilization stats can be abstracted so schedulers can use them directly
from the JT.

 - *The collecting framework of this patch can still be retained* - particularly aggregating
stats over all the TaskTrackers and per job can be retained on the JobTracker side via a _Collector_
entity embedded in JobTracker.

 - *The TaskTracker knows well about jobs/tasks/sub-processes.* The current job-based aggregation
of stats on the TaskTracker is weak because it identifies tasks corresponding to a job by
grepping for job-ids in 'ps' . This is very error-prone. This complication arises from the
fact that we are decoupling guaging from TaskTrackers into separate daemons. This can be made
concrete by doing it in TaskTracker address space itself similar to what _TaskMemoryManager_
does. TaskTracker already maintains the state we need and knows which tasks belong to this
job, and which processes belong to a task. One more reason strengthening the argument against
a new framework.

 - *The MemBasedLoadManager will still work* but by obtaining information via TaskTrackerStatus
and the _Collector_ embedded in JobTracker.

I am just trying to see if we can leverage already existing (and well tested) code.

The good news is that we can retrofit the current patch to do most of the above hardly losing
any code. Particularly most of the guaging utilities are reusable without any changes. Thoughts?

> ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory
load on TaskTracker(s)
> -----------------------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-961
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-961
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/fair-share
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: HIVE-961.patch
> Design and develop a ResouceAwareLoadManager for the FairShare scheduler that dynamically
decides how many maps/reduces to run on a particular machine based on the CPU/Memory/diskIO/network
usage in that machine.  The amount of resources currently used on each task tracker is being
fed into the ResourceAwareLoadManager in real-time via an entity that is external to Hadoop.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message