hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1608) Allow users to do speculative execution of a task manually
Date Thu, 11 Nov 2010 01:26:17 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930886#action_12930886

dhruba borthakur commented on MAPREDUCE-1608:

Amar, thanks for ur comments. For interactive workloads, this feature has worked awesomely
for us.

> As Todd/Hong have mentioned we should consider making our heuristics better. I'm not
sure this is a good idea.

I think Arun is completely missing the point here. It is fine to make heuristics better and
better, I have nothing against that. But when there is a manual-speculate-now button, it can
override the system-specified speculation heuristics for that task. To go a ste further, the
system can actually learn from the user-specified-speculation on how to automatically make
the heuristics better, i.e. crowd-sourcing to the rescue!

> Allow users to do speculative execution of a task manually
> ----------------------------------------------------------
>                 Key: MAPREDUCE-1608
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1608
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Scott Chen
>            Assignee: Scott Chen
> Speculative execution improves the latency of the job. Sometimes the job has few very
slow reducers. Spending a little more resource on speculative tasks can improve the latency
a lot. It will be nice that the users can manually select one task and force the speculative
execution on that task just like we can manually kill/fail task.
> The proposal is add link says "speculate" in taskdetails.jsp page where we do "kill/fail".
> Thoughts? 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message