hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-76) Implement speculative re-execution of reduces
Date Sat, 28 Oct 2006 03:20:20 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-76?page=comments#action_12445321 ] 
            
Owen O'Malley commented on HADOOP-76:
-------------------------------------

1. This patch doesn't apply cleanly anymore. The TaskTracker.java has a conflicting change.

2. I agree that it would be nice if FileSystem was an interface, but that is way out of scope
for this bug. 

You should derive PhasedFileSystem from FileSystem. It should have a "base" FileSystem that
does the real work and a Map<String,String> that maps "final" filenames to "temporary"
filenames. 

PhasedFileSystem will need to implement the abstract "Raw" methods from FileSystem. The read
operations, can just be sent directly to the base FileSystem. createRaw should create a temporary
name, put it in the map and create the file using the base FileSystem. The other "modification"
methods (renameRaw, deleteRaw) should throw UnsupportedOperationException.

To ensure that the files are cleaned up correctly, I'd suggest that the temporary files be
stored under:

<system dir>/<job id>/<tip id>/<task id>/<unique file id>

(in dfs clearly) so that when the job is finished, we just need to delete the job directory
to clean up any remains of failed tasks that didn't clean up properly.

there should also be:
  public void commit() throws IOException {...}
  public void abort() throws IOException {...}
  public void close() throws IOException { abort(); }

For close, it should lock the tip directory, move the files into place, put a DONE touch file
in the tip directory and delete the task id directory.

> Implement speculative re-execution of reduces
> ---------------------------------------------
>
>                 Key: HADOOP-76
>                 URL: http://issues.apache.org/jira/browse/HADOOP-76
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.1.0
>            Reporter: Doug Cutting
>         Assigned To: Sanjay Dahiya
>            Priority: Minor
>         Attachments: Hadoop-76.patch, spec_reducev.patch
>
>
> As a first step, reduce task outputs should go to temporary files which are renamed when
the task completes.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message