hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3687) Ability to pause/resume tasks
Date Tue, 11 Nov 2008 00:41:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646425#action_12646425

Matei Zaharia commented on HADOOP-3687:

The really hard challenge with pausing, in my opinion, will be how to decide when to resume
the tasks or when to kill them. It's not clear that if you pause a task on some machine, you'll
get the opportunity to run it again. In fact, maybe another machine becomes free and you'd
be better off running the task on that one. So the whole scheduling problem becomes more difficult.

Another fix that we really have to strive for is making reduces smaller, e.g. by separating
the copy phase into its own set of tasks (Joydeep has posted some comments on this in the
MapReduce 2.0 discussion).

> Ability to pause/resume tasks
> -----------------------------
>                 Key: HADOOP-3687
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3687
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Chris Smith
>            Assignee: Chris Smith
>            Priority: Minor
>         Attachments: hadoop-pausing.8.trunk.patch
> It would be nice to be able to pause (and subsequently resume) tasks that are currently
running, in order to allow tasks from higher priority jobs to execute. At present it is quite
easy for long-running tasks from low priority jobs to block a task from a newer high priority
job, and there is no way to force the execution of the high priority task without killing
the low priority jobs.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message