hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi" <knogu...@yahoo-inc.com>
Subject RE: Potential race condition (Hadoop 18.3)
Date Mon, 02 Mar 2009 20:59:12 GMT
Ryan,

If you're using getOutputPath, try replacing it with getWorkOutputPath.

http://hadoop.apache.org/core/docs/r0.18.3/api/org/apache/hadoop/mapred/
FileOutputFormat.html#getWorkOutputPath(org.apache.hadoop.mapred.JobConf
)

Koji

-----Original Message-----
From: Ryan Shih [mailto:ryan.shih@gmail.com] 
Sent: Monday, March 02, 2009 11:01 AM
To: core-user@hadoop.apache.org
Subject: Potential race condition (Hadoop 18.3)

Hi - I'm not sure yet, but I think I might be hitting a race condition
in
Hadoop 18.3. What seems to happen is that in the reduce phase, some of
my
tasks perform speculative execution but when the initial task completes
successfully, it sends a kill to the new task started. After all is said
and
done, perhaps one in every five or ten which kill their second task ends
up
with zero or truncated output.  When I code it to turn off speculative
execution, the problem goes away. Are there known race conditions that I
should be aware of around this area?

Thanks in advance,
Ryan

Mime
View raw message