hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2141) speculative execution start up condition based on completion time
Date Wed, 06 Feb 2008 16:51:08 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12566194#action_12566194

Joydeep Sen Sarma commented on HADOOP-2141:

just wanted to leave behind some recent experiences. we have started running with gap=0.9
and only turned on spec. for maps. this is working out pretty well. i see very few spec. tasks
- but they do show up when things are stuck - which is exactly what we want.

the only remaining flaw is that hasSpeculative() should not be using 'progress' - but rather
'progress/running time'. the rationale is that a late task is always going to have low progress
(even after the lag time) - so what matters is progress made relative to the run-time. if
on average tasks require 5 minutes to make 20% progress then making 5% progress in 1 minute
is perfectly fine (but today may cause spec. execution even for our conservative setting).

> speculative execution start up condition based on completion time
> -----------------------------------------------------------------
>                 Key: HADOOP-2141
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2141
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.15.0
>            Reporter: Koji Noguchi
>            Assignee: Arun C Murthy
>             Fix For: 0.17.0
> We had one job with speculative execution hang.
> 4 reduce tasks were stuck with 95% completion because of a bad disk. 
> Devaraj pointed out 
> bq . One of the conditions that must be met for launching a speculative instance of a
task is that it must be at least 20% behind the average progress, and this is not true here.
> It would be nice if speculative execution also starts up when tasks stop making progress.
> Devaraj suggested 
> bq. Maybe, we should introduce a condition for average completion time for tasks in the
speculative execution check. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message