hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-718) Support for per-phase speculative execution
Date Fri, 14 Aug 2009 04:42:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743076#action_12743076

Devaraj Das commented on MAPREDUCE-718:

bq. This will possibly put a lot more load on the JT. Are we confident this will not impact
I guess you are saying this since the system counters will be passed every heartbeat. But
the number of system counters are very few (and on top of that we can exactly control which
system counters to send if we come down to that). So hopefully, performance is not an issue.
Regarding the computation of statistics, the statistics today look at progress values, but
if we replace that by the relevant counter value for the respective phase, it shouldn't add
any extra overhead.

> Support for per-phase speculative execution
> -------------------------------------------
>                 Key: MAPREDUCE-718
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-718
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>    Affects Versions: 0.21.0
>            Reporter: Devaraj Das
>             Fix For: 0.21.0
> It would be good to have support for per-phase speculative execution where the algorithm
looks at the current phase of a task, and compares with the other tasks in the same phase
before deciding to launch a speculative task. That would have the following benefits:
> 1) Support for jobs where map tasks progresses jumps from 0% to 100%. This is true for
some jobs like randomwriter. Today, we would launch speculative tasks for such jobs (assuming
that the tasks are not making progress). But most of them would be unnecessary. 
> 2) In reality, for reduces, the three phases are quite different from each other, and
they take different times too. We should see better results when we look at per-phase speculation.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message