hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-894) dfs client protocol should allow asking for parts of the block map
Date Wed, 09 May 2007 21:46:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12494581

Doug Cutting commented on HADOOP-894:

Sameer: you're right, our current public API would not take advantage of an open with start
and length, so it may be overkill.  And in many cases we also read a file header from the
first block before we seek anyway.  Long-term, this might be a good optimization, to be able
to open a file directly at a position, without touching the first block, and to be able to
disable the reading of headers.  It would be convenient if this did not require changes to
both the protocol and to the server, but instead only on the client.  To me, open(start,length)
is a more general API that's no harder to implement than open(length), one that's future compatible.
 The client would, for now, always pass zero for 'start'.  But I wouldn't veto open(length).
 That's also a fine API and is more minimal, a good thing.

> dfs client protocol should allow asking for parts of the block map
> ------------------------------------------------------------------
>                 Key: HADOOP-894
>                 URL: https://issues.apache.org/jira/browse/HADOOP-894
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>            Reporter: Owen O'Malley
>         Assigned To: Konstantin Shvachko
>         Attachments: partialBlockList.patch, partialBlockList2.patch
> I think that the HDFS client protocol should change like:
> /** The meta-data about a file that was opened. */
> class OpenFileInfo {
>   /** the info for the first block */
>   public LocatedBlockInfo getBlockInfo();
>   public long getBlockSize();
>   public long getLength();
> }
> interface ClientProtocol extends VersionedProtocol {
>   public OpenFileInfo open(String name) throws IOException;
>   /** get block info for any range of blocks */
>   public LocatedBlockInfo[] getBlockInfo(String name, int blockOffset, int blockLength)
throws IOException;
> }
> so that the client can decide how much block info to request and when. Currently, when
the file is opened or an error occurs, the entire block list is requested and sent.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message