hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1542) Speculative execution used when property set to false
Date Thu, 28 Jun 2007 06:13:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12508739
] 

Devaraj Das commented on HADOOP-1542:
-------------------------------------

It looks like the JobTracker is ignoring the mapred-default.xml config items and hence the
speculative execution setting in mapred-default.xml is not reflected. However, on the tasktrackers,
the mapred-default.xml indeed overrides the config in hadoop-site/hadoop-default.xml, and
hence sees speculative execution switched off.
This could be the cause of the dfs file-lost problem. Here's the theory (not yet validated
from the source code): When the maps tries to create files on dfs, they try to the create
the "final" files (as opposed to the speculative case where the output path for the files
would point to task specific directories). Hence the spec instance of a map gets the AlreadyBeingCreatedException.
Finally the JT, which thinks that spec exec is turned on, tries to rename the empty file path
to its final destination and that overwrites the real file that the task originally created.

> Speculative execution used when property set to false
> -----------------------------------------------------
>
>                 Key: HADOOP-1542
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1542
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Nigel Daley
>         Attachments: jobtracker.log, namenode.log
>
>
> Speculative execution is now on by default.  When running TestDFSIO, I set speculative
execution off in my mapred-default.xml since this test has maps that create files in DFS (side-effects).
 
> However, it seems that speculative tasks get started even though I have set speculation
off.  I'll attached the NN and JT logs.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message