hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Chen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1608) Allow users to do speculative execution of a task manually
Date Fri, 19 Mar 2010 01:22:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847189#action_12847189

Scott Chen commented on MAPREDUCE-1608:

Thanks, Tood. That's a good point. We should also work on making the tuning knobs better.

The reason why I am proposing this is that one of our user ask me if there is speculative
execution in a single reducer case.
Our speculation policy is based on comparing a task to other tasks (or some average behavior).
So in this case the speculative execution will not be triggered. But the human knows that
this task is slow.

The tuning knobs can be very good but it is hard to make it perfect.
It is good to have some control over the task when we need it (just like Kill/Fail task).

> Allow users to do speculative execution of a task manually
> ----------------------------------------------------------
>                 Key: MAPREDUCE-1608
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1608
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Scott Chen
>            Assignee: Scott Chen
> Speculative execution improves the latency of the job. Sometimes the job has few very
slow reducers. Spending a little more resource on speculative tasks can improve the latency
a lot. It will be nice that the users can manually select one task and force the speculative
execution on that task just like we can manually kill/fail task.
> The proposal is add link says "speculate" in taskdetails.jsp page where we do "kill/fail".
> Thoughts? 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message