hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dennis Kubes (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-2391) Speculative Execution race condition with output paths
Date Sun, 09 Dec 2007 06:12:43 GMT
Speculative Execution race condition with output paths
------------------------------------------------------

                 Key: HADOOP-2391
                 URL: https://issues.apache.org/jira/browse/HADOOP-2391
             Project: Hadoop
          Issue Type: Bug
         Environment: all
            Reporter: Dennis Kubes
            Assignee: Dennis Kubes


I am tracking a problem where when speculative execution is enabled, there is a race condition
when trying to read output paths from a previously completed job.  More specifically when
reduce tasks run their output is put into a working directory under the task name until the
task in completed.  The directory name is something like workdir/_taskid.  Upon completion
the output get moved into workdir.  Regular tasks are checked for this move and not considered
completed until this move is made.  I have not verified it but all indications point to speculative
tasks NOT having this same check for completion and more importantly removal when killed.
 So what we end up with when trying to read the output of previous tasks with speculative
execution enabled is the possibility that previous workdir/_taskid will be present when the
output directory is read by a chained job.  Here is an error when supports my theory:

Generator: org.apache.hadoop.ipc.RemoteException: java.io.IOException: Cannot open filename
/u01/hadoop/mapred/temp/generate-temp-1197104928603/_task_200712080949_0005_r_000014_1
        at org.apache.hadoop.dfs.NameNode.open(NameNode.java:234)
        at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:389)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:644)
        at org.apache.hadoop.ipc.Client.call(Client.java:507)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:186)
        at org.apache.hadoop.dfs.$Proxy0.open(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at org.apache.hadoop.dfs.$Proxy0.open(Unknown Source)
        at org.apache.hadoop.dfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:839)
        at org.apache.hadoop.dfs.DFSClient$DFSInputStream.<init>(DFSClient.java:831)
        at org.apache.hadoop.dfs.DFSClient.open(DFSClient.java:263)
        at org.apache.hadoop.dfs.DistributedFileSystem.open(DistributedFileSystem.java:114)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1356)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1349)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1344)
        at org.apache.hadoop.mapred.SequenceFileOutputFormat.getReaders(SequenceFileOutputFormat.java:87)
        at org.apache.nutch.crawl.Generator.generate(Generator.java:429)
        at org.apache.nutch.crawl.Generator.run(Generator.java:563)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:54)
        at org.apache.nutch.crawl.Generator.main(Generator.java:526)

I will continue to research this and post as I make progress on tracking down this bug.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message