hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hong Tang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1608) Allow users to do speculative execution of a task manually
Date Thu, 11 Nov 2010 22:18:18 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931233#action_12931233

Hong Tang commented on MAPREDUCE-1608:

bq. But when there is a manual-speculate-now button, it can override the system-specified
speculation heuristics for that task. To go a ste further, the system can actually learn from
the user-specified-speculation on how to automatically make the heuristics better, i.e. crowd-sourcing
to the rescue!

I agree that knowledgeable users have more information about their programs and probably can
speculate more accurately. Such info may also be used for the verification (future tuning)
of speculative execution algorithms (e.g. through simulation). So here is my +1 on the idea.

> Allow users to do speculative execution of a task manually
> ----------------------------------------------------------
>                 Key: MAPREDUCE-1608
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1608
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Scott Chen
>            Assignee: Scott Chen
> Speculative execution improves the latency of the job. Sometimes the job has few very
slow reducers. Spending a little more resource on speculative tasks can improve the latency
a lot. It will be nice that the users can manually select one task and force the speculative
execution on that task just like we can manually kill/fail task.
> The proposal is add link says "speculate" in taskdetails.jsp page where we do "kill/fail".
> Thoughts? 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message