hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hong Tang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1218) Collecting cpu and memory usage for TaskTrackers
Date Thu, 19 Nov 2009 04:42:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779829#action_12779829
] 

Hong Tang commented on MAPREDUCE-1218:
--------------------------------------

I realized you are looking for some quick solutions. So I am fine with your suggestions.

>From a long term perspective, I think we ought to think more on the overall architecture
of hadoop:

In some cluster management systems I know of, typically load management and service discovery
are structured as a standalone layer, and computation framework and storage systems like MR,
HDFS would be services built on top of that. There are many advantages of such a design over
the current architecture, two of which on top of head are:
- the load information may be shared across multiple tenants that share the same resources,
and they can coordinate on load balance objectives.
- code is more modular and easier to maintain

There are also a lot of researches on dispersing load information across nodes and using the
load information wisely, particularly in situations where load information fluctuate quite
frequently (so once every 20s heartbeat could be too coarse-grained). There are a series of
papers in these areas from the research group where I did my Ph.D.: http://www.cs.ucsb.edu/projects/neptune/

> Collecting cpu and memory usage for TaskTrackers
> ------------------------------------------------
>
>                 Key: MAPREDUCE-1218
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1218
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>         Environment: linux
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>
> The information can be used for resource aware scheduling.
> Note that this is related to MAPREDUCE-220. There the per task resource information is
collected.
> This one collects the per machine information.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message