hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-1542) Incorrect task/tip being scheduled (looks like speculative execution)
Date Thu, 28 Jun 2007 22:48:04 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Owen O'Malley updated HADOOP-1542:
----------------------------------

        Fix Version/s: 0.14.0
             Assignee: Owen O'Malley  (was: Devaraj Das)
          Description: 
The change in HADOOP-1440 broke map/reduce by breaking the assumption that Task.getPartition()
corresponded to the JobInProgress.map[] order. 

Currently JobInProgress.findNewTask uses Task.getPartition as the index of the map to run.
This can be a completely different tip, which will cause incorrect tasks to be run, including
duplicates of tasks that are already running.

  was:
Speculative execution is now on by default.  When running TestDFSIO, I set speculative execution
off in my mapred-default.xml since this test has maps that create files in DFS (side-effects).
 

However, it seems that speculative tasks get started even though I have set speculation off.
 I'll attached the NN and JT logs.

    Affects Version/s: 0.14.0
              Summary: Incorrect task/tip being scheduled (looks like speculative execution)
 (was: Speculative execution used when property set to false)

> Incorrect task/tip being scheduled (looks like speculative execution)
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-1542
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1542
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.14.0
>            Reporter: Nigel Daley
>            Assignee: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: jobtracker.log, namenode.log
>
>
> The change in HADOOP-1440 broke map/reduce by breaking the assumption that Task.getPartition()
corresponded to the JobInProgress.map[] order. 
> Currently JobInProgress.findNewTask uses Task.getPartition as the index of the map to
run. This can be a completely different tip, which will cause incorrect tasks to be run, including
duplicates of tasks that are already running.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message