hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1259) DFS should enforce block size is a multiple of io.bytes.per.checksum
Date Fri, 13 Apr 2007 19:46:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12488766

Doug Cutting commented on HADOOP-1259:

> post-HADOOP-1134-upgrade, such a mismatch will cause similar issue, for e.g., to join
two blocks, we will need to re-checksum the entire second block

When do we anticipate that we'll need to join blocks?  We might, to de-fragment files that
have been appended to many times.  But in that case the blocks will likely be on separate
datanodes, so re-checksumming would incur no cost.  De-fragmenting could probably be done
fairly efficiently by a map-reduce task in user code, by simply copying a file to a new, temporary
name, then renaming it back to its original name.  So I don't yet see block joining as an
argument for this.

> DFS should enforce block size is a multiple of io.bytes.per.checksum 
> ---------------------------------------------------------------------
>                 Key: HADOOP-1259
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1259
>             Project: Hadoop
>          Issue Type: Improvement
>            Reporter: Raghu Angadi
> DFSClient currently does not enforce that dfs.block.size is a multiple io.bytes.per.checksum.
This not really problem currently but can future upgrades like HADOOP-1134 (see one of the
comments http://issues.apache.org/jira/browse/HADOOP-1134#action_12488542 there). 
> I propose DFSClient should fail loudly and ask the user politely to change the config
to meet this conidtion. Of course we will change the documentation for dfs.block.size also.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message