hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-76) Implement speculative re-execution of reduces
Date Fri, 27 Oct 2006 00:07:19 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-76?page=comments#action_12445048 ] 
Owen O'Malley commented on HADOOP-76:

The PhasedRecordWriter won't handle all of the cases, because RecordWriters can write multiple
files. Furthermore, they are user code and it would be better to minimize required changes
to them.

A better approach would be to have a PhasedFileSystem that takes a base FileSystem and use
that to commit/abort the changes. Then the framework could pass the PhasedFileSystem to the
createRecordReader call and it would catch all of the files that the RecordWriter created.
When the PhasedFileSystem gets a create call, it creates it in the base FileSystem with a
mutated name. When the changes are commited, the files are all renamed. If the changes are
aborted, the mutated filenames are deleted.

> Implement speculative re-execution of reduces
> ---------------------------------------------
>                 Key: HADOOP-76
>                 URL: http://issues.apache.org/jira/browse/HADOOP-76
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.1.0
>            Reporter: Doug Cutting
>         Assigned To: Sanjay Dahiya
>            Priority: Minor
>         Attachments: Hadoop-76.patch, spec_reducev.patch
> As a first step, reduce task outputs should go to temporary files which are renamed when
the task completes.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message