hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-212) allow changes to dfs block size
Date Fri, 12 May 2006 17:30:09 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-212?page=comments#action_12383223 ] 

Owen O'Malley commented on HADOOP-212:
--------------------------------------

As you point out, it is possible to just make configuration variable and use it everywhere.
The problem is that you become very sensititve to differences in the configuration between
nodes. It seemed less error prone to leave the client in charge of block size and consistently
use their setting.

Under the hood, dfs currently supports variable block sizes within a file, but I certainly
do _not_ want to expose that to the user-visible APIs.

> allow changes to dfs block size
> -------------------------------
>
>          Key: HADOOP-212
>          URL: http://issues.apache.org/jira/browse/HADOOP-212
>      Project: Hadoop
>         Type: Improvement

>   Components: dfs
>     Versions: 0.2
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>     Priority: Critical
>      Fix For: 0.3

>
> Trying to change the DFS block size, led the realization that the 32,000,000 was hard
coded into the source code. I propose:
>   1. Change the default block size to 64 * 1024 * 1024.
>   2. Add the config variable dfs.block.size that sets the default block size.
>   3. Add a parameter to the FileSystem, DFSClient, and ClientProtocol create method that
let's the user control the block size.
>   4. Rename the FileSystem.getBlockSize to getDefaultBlockSize.
>   5. Add a new method to FileSytem.getBlockSize that takes a pathname.
>   6. Use long for the block size in the API, which is what was used before. However,
the implementation will not work if block size is set bigger than 2**31.
>   7. Have the InputFormatBase use the blocksize of each file to determine the split size.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message