hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-894) dfs client protocol should allow asking for parts of the block map
Date Tue, 15 May 2007 23:03:16 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Konstantin Shvachko updated HADOOP-894:

    Attachment: partialBlockList3.patch

In this patch:
- open takes three parameters open(src,  offset, length)
- there is an undocumented config parameter "dfs.read.prefetch.size" that defines the range
within which 
we want all block locations to be fetch from the name-node during the open call.
- I kept 10*defaultBlockSize as the default, because 2 vs 10 does not improve much communication
or name-node
performance, but in most cases 10 will be ALL blocks for the majority of files.
- Implemented block location caching for reads and preads.
- Included more test cases in TestPread

> dfs client protocol should allow asking for parts of the block map
> ------------------------------------------------------------------
>                 Key: HADOOP-894
>                 URL: https://issues.apache.org/jira/browse/HADOOP-894
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>            Reporter: Owen O'Malley
>         Assigned To: Konstantin Shvachko
>         Attachments: partialBlockList.patch, partialBlockList2.patch, partialBlockList3.patch
> I think that the HDFS client protocol should change like:
> /** The meta-data about a file that was opened. */
> class OpenFileInfo {
>   /** the info for the first block */
>   public LocatedBlockInfo getBlockInfo();
>   public long getBlockSize();
>   public long getLength();
> }
> interface ClientProtocol extends VersionedProtocol {
>   public OpenFileInfo open(String name) throws IOException;
>   /** get block info for any range of blocks */
>   public LocatedBlockInfo[] getBlockInfo(String name, int blockOffset, int blockLength)
throws IOException;
> }
> so that the client can decide how much block info to request and when. Currently, when
the file is opened or an error occurs, the entire block list is requested and sent.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message