hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-451) Add a Split interface
Date Tue, 26 Sep 2006 18:24:52 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-451?page=comments#action_12437900 ] 
            
Doug Cutting commented on HADOOP-451:
-------------------------------------

I like option 3 best -- add a RecordReader.getProgress() method, replacing its getPos() method
with something abstract and file-independent.

I also wonder whether we then really need Split.getCost().  We could instead promise to process
splits in the order they're generated, so jobs could still control the order if they liked.

> Add a Split interface
> ---------------------
>
>                 Key: HADOOP-451
>                 URL: http://issues.apache.org/jira/browse/HADOOP-451
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Doug Cutting
>
> The InputFormat interface has a method:
> FileSplit[] getSplits();
> This should change to:
> Split[] getSplits();
> The Split interface would look like:
> public interface Split extends Writable {
>   /** Returns a list of hosts that contain this split.
>        This is only used to optimize task placement, so this may be empty. */
>   String[] getLocations(FileSystem fs);
>   /** The relative, estimated cost of operating on this.  Typically the size of the data
in the split.
>        Used to prioritize tasks in a job (high-cost tasks are run first).  */
>    long getCost();
> }

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message