hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-961) ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory load on TaskTracker(s)
Date Thu, 15 Oct 2009 11:32:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766025#action_12766025

Vinod K V commented on MAPREDUCE-961:

I think you got my point.

bq. The reason why we built them as separate daemons is mainly because we want this to run
on multiple map-reduce clusters as Dhruba mentioned.
bq. will definitely study how to put these daemons inside TT and JT. I think one possibility
is that we build them inside TT and JT but still provide the RPC interface in Collector.
bq. If we need information on multiple clusters, we can go to the corresponding Collectors
and get them via RPC.

That sounds a slightly different use-case to me. Metrics api can be used for this.

Just curious: how do you intend to use it? Currently scheduler is very tightly coupled with
a single cluster/JobTracker. Information exposed by multiple clusters is currently unusable
by any single cluster unless you have some external components. If indeed you have external
components outside of mapred, metrics api seems the correct tool. Thoughts?

bq. Also, at this stage, it is easy to test these daemons without the dependency on JT or
TT. We can easily change/restart these daemons without affecting the map-reduce cluster.
Perhaps. But I think eventually we should move inside the framework. In any case, for a clean
design, we can still factor them out to well defined classes and so. Once that is done, if
ever we want to move them out into separate daemons, it won't be infinitely complex.

> ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory
load on TaskTracker(s)
> -----------------------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-961
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-961
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/fair-share
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: HIVE-961.patch
> Design and develop a ResouceAwareLoadManager for the FairShare scheduler that dynamically
decides how many maps/reduces to run on a particular machine based on the CPU/Memory/diskIO/network
usage in that machine.  The amount of resources currently used on each task tracker is being
fed into the ResourceAwareLoadManager in real-time via an entity that is external to Hadoop.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message