hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nandan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2653) dynamic map slots (in addition to predifined) on each node which allows to execute cpu intensive jobs along with memory intensive jobs thereby reducing wastage of cpu cycles
Date Wed, 24 Aug 2011 05:58:29 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090015#comment-13090015
] 

nandan commented on MAPREDUCE-2653:
-----------------------------------

In response to Allen Wittenauer's question: 
How does this method work when the tasks are IO intensive? 

Monitoring system on every TT categorizes and stores each task it runs, into CPU-Intensive
and CPU-NonIntensive lists (this includes Memory as well IO intensive tasks) and generates
job request by selecting jobs from these lists one by one alternately, considering current
cpu-idle time and cpu utilization of the task. Request consists of list of jobs whose map
tasks TT can run as extra tasks. This request is submitted to JT through heartbeat, which
processes jobs from the request one by one. 

So currently I am treating IO and Memory processes as same.

> dynamic map slots (in addition to predifined) on each node which allows to execute cpu
intensive jobs along with memory intensive jobs thereby reducing wastage of cpu cycles
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2653
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2653
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker, tasktracker
>    Affects Versions: 0.20.203.0
>         Environment: linux
>            Reporter: nandan
>              Labels: map, scheduler, tasks
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> I have introduced process monitoring system inside tasktracker, which analyses the cpu
and memory utilization of each map task and allows me to increase/decrease maximum number
of map slots dynamically on each node. With this I can combine cpu intensive jobs along with
memory intensive jobs, thereby reducing the cpu idle time.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message