hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-894) dfs client protocol should allow asking for parts of the block map
Date Thu, 10 May 2007 00:43:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12494608

Konstantin Shvachko commented on HADOOP-894:

Yes. LocatedBlocks contains file length and a List of block locations.
I initially implemented open(src, length) because it is more general, and deprecated old open(src).
Dhruba finds it "not very intuitive" and Sameer says it does not "add a lot of value".

I cannot implement open(start,length) with the start > 0 right now, because in order to
do that I will
need to write an offset-to-block map for cached blocks in the client. I was planning to do
it in the next
iteration, but it was supposed to be used mostly in pread() that is for getBlockLocations(),
not in open().

I don't see how we can benefit from introducing the start parameter, but I definitely support
adding length.
So currently it's a tie 2:2. We need more votes to resolve the issue.

> dfs client protocol should allow asking for parts of the block map
> ------------------------------------------------------------------
>                 Key: HADOOP-894
>                 URL: https://issues.apache.org/jira/browse/HADOOP-894
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>            Reporter: Owen O'Malley
>         Assigned To: Konstantin Shvachko
>         Attachments: partialBlockList.patch, partialBlockList2.patch
> I think that the HDFS client protocol should change like:
> /** The meta-data about a file that was opened. */
> class OpenFileInfo {
>   /** the info for the first block */
>   public LocatedBlockInfo getBlockInfo();
>   public long getBlockSize();
>   public long getLength();
> }
> interface ClientProtocol extends VersionedProtocol {
>   public OpenFileInfo open(String name) throws IOException;
>   /** get block info for any range of blocks */
>   public LocatedBlockInfo[] getBlockInfo(String name, int blockOffset, int blockLength)
throws IOException;
> }
> so that the client can decide how much block info to request and when. Currently, when
the file is opened or an error occurs, the entire block list is requested and sent.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message