hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3738) Spawning tasks faster
Date Thu, 10 Jul 2008 03:17:31 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612363#action_12612363
] 

Devaraj Das commented on HADOOP-3738:
-------------------------------------

I am tending to think that we should probably invest the energy in the other approach. That's
HADOOP-3136.

> Spawning tasks faster
> ---------------------
>
>                 Key: HADOOP-3738
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3738
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Spyros Blanas
>            Priority: Minor
>         Attachments: dynamic_heartbeat.patch
>
>
> In the current implementation, tasks are assigned to tasktrackers by adding an appropriate
action to the heartbeat response list. Each heartbeat response can start one task. As the
minimum interval between heartbeats is 5 sec (by default), if the nodes are strong machines
(say, each node has 10 task "slots") and the cluster is idle, this means that some tasks are
spawned after some time (in our example, the last task will be spawned after 45 seconds).
> This can be significantly improve the end-to-end execution time if most jobs are finished
in the order of minutes.
> The patch I attach requests from each TaskTracker to reply in 1/5th of the regular heartbeat
interval time if it was assigned a task in this round, making spawning of multiple tasks much
more efficient.
> A better approach would be to have each TaskTracker report the number of free slots it
has (instead of only if it can accept more work or not) and have the JobTracker push the appropriate
number of tasks in one response, but this will require changes in the current communication
protocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message