hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-212) allow changes to dfs block size
Date Mon, 15 May 2006 18:08:06 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-212?page=all ]

Owen O'Malley updated HADOOP-212:
---------------------------------

    Attachment: dfs-blocksize.patch

Ok, here is the patch.

Changes dfs block size from a compile time constant to a parameter that is set when a file
is created.

1. FileSystem.getBlockSize becomes getDefaultBlockSize
2. A new method FileSystem.getBlockSize(path) finds the blocksize of a file.
3. Block size is added to FileSystem.create
4. InputFormatBase uses the block size of each file rather than the global constant.
5. I followed the convention of using DfsPath to cache meta information values associatied
with the dfs file.
6. FileUnderConstruction records the block size
7. Removed check to make sure that the block size was shorter than the global value.
8. Add a new value

> allow changes to dfs block size
> -------------------------------
>
>          Key: HADOOP-212
>          URL: http://issues.apache.org/jira/browse/HADOOP-212
>      Project: Hadoop
>         Type: Improvement

>   Components: dfs
>     Versions: 0.2
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>     Priority: Critical
>      Fix For: 0.3
>  Attachments: dfs-blocksize.patch
>
> Trying to change the DFS block size, led the realization that the 32,000,000 was hard
coded into the source code. I propose:
>   1. Change the default block size to 64 * 1024 * 1024.
>   2. Add the config variable dfs.block.size that sets the default block size.
>   3. Add a parameter to the FileSystem, DFSClient, and ClientProtocol create method that
let's the user control the block size.
>   4. Rename the FileSystem.getBlockSize to getDefaultBlockSize.
>   5. Add a new method to FileSytem.getBlockSize that takes a pathname.
>   6. Use long for the block size in the API, which is what was used before. However,
the implementation will not work if block size is set bigger than 2**31.
>   7. Have the InputFormatBase use the blocksize of each file to determine the split size.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message