hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "eric baldeschwieler (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-723) Race condition exists in the method MapOutputLocation.getFile
Date Wed, 15 Nov 2006 19:55:39 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-723?page=comments#action_12450178 ] 
eric baldeschwieler commented on HADOOP-723:

You might want to use a temp file and rename strategy to make sure:

a) you only delete files you own
b) it is clear that a file was completely transfered by inspection.
c) temp files can be easily cleaned up in some conditions (restart say)

> Race condition exists in the method MapOutputLocation.getFile
> -------------------------------------------------------------
>                 Key: HADOOP-723
>                 URL: http://issues.apache.org/jira/browse/HADOOP-723
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
> There seems to be a race condition in the way the Reduces copy the map output files from
the Maps. If a copier is blocked in the connect method (in the beginning of the method MapOutputLocation.getFile)
to a Jetty on a Map, and the MapCopyLeaseChecker detects that the copier was idle for too
long, it will go ahead and issue a interrupt (read 'kill') to this thread and create a new
Copier thread. However, the copier, currently blocked trying to connect to Jetty on a Map,
doesn't actually get killed until the connect timeout expires and as soon as the connect comes
out (with an IOException), it will delete the map output file which actually could have been
(successfully) created by the new Copier thread. This leads to the Sort phase for that reducer
failing with a FileNotFoundException.
> One simple way to fix this is to not delete the file if the file was not created within
this getFile method.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message