hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Milind Bhandarkar (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-212) allow changes to dfs block size
Date Mon, 15 May 2006 20:25:07 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-212?page=comments#action_12402410 ] 

Milind Bhandarkar commented on HADOOP-212:
------------------------------------------

Doug, the problem is occuring much earlier before NPE:

060515 123400 DIR* FSDirectory.mkdirs: failed to create directory /srcdat

Looks like your DFS config has a root directory not writable for you. Because the test is
trying to create dfs://localhost:65314/srcdat. I am using org.apache.hadoop.dfs.MiniDFSCluster
for testing copying across dfs.


> allow changes to dfs block size
> -------------------------------
>
>          Key: HADOOP-212
>          URL: http://issues.apache.org/jira/browse/HADOOP-212
>      Project: Hadoop
>         Type: Improvement

>   Components: dfs
>     Versions: 0.2
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>     Priority: Critical
>      Fix For: 0.3
>  Attachments: TEST-org.apache.hadoop.fs.TestCopyFiles.txt, dfs-blocksize.patch
>
> Trying to change the DFS block size, led the realization that the 32,000,000 was hard
coded into the source code. I propose:
>   1. Change the default block size to 64 * 1024 * 1024.
>   2. Add the config variable dfs.block.size that sets the default block size.
>   3. Add a parameter to the FileSystem, DFSClient, and ClientProtocol create method that
let's the user control the block size.
>   4. Rename the FileSystem.getBlockSize to getDefaultBlockSize.
>   5. Add a new method to FileSytem.getBlockSize that takes a pathname.
>   6. Use long for the block size in the API, which is what was used before. However,
the implementation will not work if block size is set bigger than 2**31.
>   7. Have the InputFormatBase use the blocksize of each file to determine the split size.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message