hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Spyros Blanas (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3738) Spawning tasks faster
Date Thu, 10 Jul 2008 01:53:31 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Spyros Blanas updated HADOOP-3738:

    Attachment: dynamic_heartbeat.patch

> Spawning tasks faster
> ---------------------
>                 Key: HADOOP-3738
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3738
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Spyros Blanas
>            Priority: Minor
>         Attachments: dynamic_heartbeat.patch
> In the current implementation, tasks are assigned to tasktrackers by adding an appropriate
action to the heartbeat response list. Each heartbeat response can start one task. As the
minimum interval between heartbeats is 5 sec (by default), if the nodes are strong machines
(say, each node has 10 task "slots") and the cluster is idle, this means that some tasks are
spawned after some time (in our example, the last task will be spawned after 45 seconds).
> This can be significantly improve the end-to-end execution time if most jobs are finished
in the order of minutes.
> The patch I attach requests from each TaskTracker to reply in 1/5th of the regular heartbeat
interval time if it was assigned a task in this round, making spawning of multiple tasks much
more efficient.
> A better approach would be to have each TaskTracker report the number of free slots it
has (instead of only if it can accept more work or not) and have the JobTracker push the appropriate
number of tasks in one response, but this will require changes in the current communication

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message