hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-1778) The shuffle keeps the ReduceTask locked while doing a FileSystem.rename leading to task timeouts
Date Fri, 24 Aug 2007 18:33:30 GMT
The shuffle keeps the ReduceTask locked while doing a FileSystem.rename leading to task timeouts
------------------------------------------------------------------------------------------------

                 Key: HADOOP-1778
                 URL: https://issues.apache.org/jira/browse/HADOOP-1778
             Project: Hadoop
          Issue Type: Bug
          Components: mapred
    Affects Versions: 0.14.0
            Reporter: Owen O'Malley
            Assignee: Owen O'Malley


The shuffle in ReduceTask.ReduceCopier.MapOutputCopier.copyOutput locks the entire ReduceTask
while doing a FileSystem.rename operation. Unfortunately the RawLocalFileSystem implements
rename as a copy and delete, which can take a long time. As a result the reduce is being killed
as not reporting progress for 10 minutes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message