hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1296) Improve interface to FileSystem.getFileCacheHints
Date Wed, 25 Apr 2007 21:15:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491776
] 

Owen O'Malley commented on HADOOP-1296:
---------------------------------------

Maybe I was just trying to be too clever. :)

Since the BlockInfo's come in a list of presumably sequential blocks, the start of each block
lets you determine how long the previous block is except the last. Even the last is bounded
by the length that the user passed in. 

It would work equally well to provide the lengths instead of the sizes as long as the first
block in the returned list was only reported as a length from the given offset. To me the
two solutions seemed equivalent, except that the starts didn't need to be adjusted based on
the user input and the lengths did. What do other people think?

> Improve interface to FileSystem.getFileCacheHints
> -------------------------------------------------
>
>                 Key: HADOOP-1296
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1296
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: fs
>            Reporter: Owen O'Malley
>         Assigned To: dhruba borthakur
>
> The FileSystem interface provides a very limited interface for finding the location of
the data. The current method looks like:
> String[][] getFileCacheHints(Path file, long start, long len) throws IOException
> which returns a list of "block info" where the block info consists of a list host names.
Because the hints don't include the information about where the block boundaries are, map/reduce
is required to call the name node for each split. I'd propose that we fix the naming a bit
and make it:
> public class BlockInfo extends Writable {
>   public long getStart();
>   public String[] getHosts();
> }
> BlockInfo[] getFileHints(Path file, long start, long len) throws IOException;
> So that map/reduce can query about the entire file and get the locations in a single
call.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message