hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-717) Fix some corner case issues in speculative execution (post hadoop-2141)
Date Fri, 17 Jul 2009 08:51:14 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Devaraj Das updated MAPREDUCE-717:

    Attachment: 717.patch

Apart from what I mentioned in the jira description, here are a few other fixes:
1) Changed the speculative cap to take into account the tasktype (map and reduce). In the
HADOOP-2141 patch, it is a global count for maps and reduces. The problem I encountered was
that in some cases, the reduce tasks would take up the slots towards the beginning of the
job, and the last few maps wouldn't get any speculative slot. 
2) Changed the dispatch time for a task to be per task-attempt based. In the HADOOP-2141 patch,
it is per task based.

All the unit tests & test-patch passed.

> Fix some corner case issues in speculative execution (post hadoop-2141)
> -----------------------------------------------------------------------
>                 Key: MAPREDUCE-717
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-717
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>    Affects Versions: 0.21.0
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.21.0
>         Attachments: 717.patch, 717.patch
> Some corner case issues can be fixed:
> 1) Setup task should not add anything to the job statistics (since they are really fast
and might affect the statistics of a job with few tasks)
> 2) The statistics computations should be guarded for cases where things like sumOfSquares
could become less than zero (due to rounding errors mostly).
> 3) The method TaskInProgress.getCurrentProgressRate() should take into account the COMMIT_PENDING
> 4) The testcase TestSpeculativeExecution.testTaskLATEScheduling could be made more robust

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message