hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-679) Block receiver unexpectedly throws an IOException complaining mismatched checksum
Date Wed, 07 Oct 2009 18:41:31 GMT

    [ https://issues.apache.org/jira/browse/HDFS-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763179#action_12763179

Hairong Kuang commented on HDFS-679:

It turns out there is also a bug at the DataNode side that handles appending to partial checksum
chunk incorrectly. Let me try to explain. 

Suppose that a file has one byte 'a'. It is open for append. The client writes one byte 'b'
and calls hflush. A packet containing 'b" gets pushed to a datanode. The datanode reads 'a'
and its checksum from disk, validating its on-disk data, then recaculates the checksum for
"ab", writes 'b' to disk, and overwrites the checksum for the chunk. So far so good. However,
when the client writes another byte "c" and calls hflush again, the client send a packet "bc"
not "c" to the datanode side. When the datanode reads "a" and checksums from the disk, checksum
validation fails because the checksum on disk is not the checksum for "a" but for "ab". The
write fails at the following lines in BlockReceiver#computePartialChunkCrc:

    if (partialCrc.getValue() != FSInputChecker.checksum2long(crcbuf)) {
      String msg = "Partial CRC " + partialCrc.getValue() +
                   " does not match value computed the " +
                   " last time file was closed " +
      throw new IOException(msg);

> Block receiver unexpectedly throws an IOException complaining mismatched checksum
> ---------------------------------------------------------------------------------
>                 Key: HDFS-679
>                 URL: https://issues.apache.org/jira/browse/HDFS-679
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
> When I run TestClientProtocolForPipelineRecovery, I always see that the block receiver
throws IOException complaining about mismatched checksum when receiving the last data packet.
It turns out the checksum of last packet was unexpectedly set to be zero.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message