hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kousuke Saruta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5761) DataNode fails to validate integrity for checksum type NULL when DataNode recovers
Date Thu, 16 Jan 2014 18:35:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873719#comment-13873719

Kousuke Saruta commented on HDFS-5761:

Thanks for your comment, Uma.
At first, I thought same as you.
I thought it's good to branch the logic depending on whether checksum type is NULL or not.
But, on second thought, BlockPoolSlice should not have logic which depends specific checksum
How to verify is responsibility of each checksum algorithm.

> DataNode fails to validate integrity for checksum type NULL when DataNode recovers 
> -----------------------------------------------------------------------------------
>                 Key: HDFS-5761
>                 URL: https://issues.apache.org/jira/browse/HDFS-5761
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 3.0.0
>            Reporter: Kousuke Saruta
>            Assignee: Kousuke Saruta
>         Attachments: HDFS-5761.patch
> When DataNode is down during writing blocks, the blocks are not filinalized and the next
time DataNode recovers, integrity validation will run.
> But if we use NULL for checksum algorithm (we can set NULL to dfs.checksum.type), DataNode
will fail to validate integrity and cannot be up. 
> The cause is in BlockPoolSlice#validateIntegrity.
> In the method, there is following code.
> {code}
> long numChunks = Math.min(
>           (blockFileLen + bytesPerChecksum - 1)/bytesPerChecksum, 
>           (metaFileLen - crcHeaderLen)/checksumSize);
> {code}
> When we choose NULL checksum, checksumSize is 0 so ArithmeticException will be thrown
and DataNode cannot be up.

This message was sent by Atlassian JIRA

View raw message