hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Dahiya (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-76) Implement speculative re-execution of reduces
Date Fri, 27 Oct 2006 12:55:19 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-76?page=comments#action_12445173 ] 
Sanjay Dahiya commented on HADOOP-76:

Looking at FileSystem implementation here is a concern 

FileSystem is an abstract class, extended by Local/Distributed FileSystem. ideal way to have
a PhasedFileSystem would be if FileSystem was an interface, we could implement it and take
a FileSystem in the constructor of PhasedFIleSystem. some thing like  - 
PhasedFileSystem implements FileSystem{
   public PhasedFileSystem(FileSysten fs);
   // channel all methods to fs 
  // implement commit() / abort()
In this case we could use Phased functionality with both Local/Distributed FileSystem, still
maintaining interface compatibility. Doing this with base abstract class will cause some dummy
objects. as in next option 

Next option is to have something like 

PhasedFileSystem extends FileSystem{
   private FileSystem fs ; 
   protected PhasedFileSystem(FileSysten fs) { 
    super(fs.getConf()); // not used, just dummy for base class creation
   	this.fs = fs ; 
   protected PhasedFileSystem(conf){ super(conf); throw NotSupportedException; } 
   public static getNamed(...) { 
     return new PhasedFileSystem(FileSystem.getNamed..) ;
   //  All other methods channel calls to fs
   commit() ;

Last option is to add extra methods to FileSystem itself, which will not be used in most cases
but will be available in derived FileSystems. This doesnt sound too good. 
Option 2 is something that will work well for us, even though its not a good design.


> Implement speculative re-execution of reduces
> ---------------------------------------------
>                 Key: HADOOP-76
>                 URL: http://issues.apache.org/jira/browse/HADOOP-76
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.1.0
>            Reporter: Doug Cutting
>         Assigned To: Sanjay Dahiya
>            Priority: Minor
>         Attachments: Hadoop-76.patch, spec_reducev.patch
> As a first step, reduce task outputs should go to temporary files which are renamed when
the task completes.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message