hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-679) Block receiver unexpectedly throws an IOException complaining mismatched checksum
Date Tue, 06 Oct 2009 23:50:31 GMT

    [ https://issues.apache.org/jira/browse/HDFS-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762818#action_12762818
] 

Hairong Kuang commented on HDFS-679:
------------------------------------

The problem is caused by incorrect handling of checksum when appending to a partial chunk.
Specifically the following code in DFSClient#DFSOutputStream#writeChunk:
{noformat}
        // If this was the first write after reopening a file, then the above
        // write filled up any partial chunk. Tell the summer to generate full 
        // crc chunks from now on.
        if (appendChunk) {
          appendChunk = false;
          resetChecksumChunk(bytesPerChecksum);
        }
{noformat}
The code blindly assumes that the first packet after reopening a file for append fills up
the partial chunk. But in case of hflush, a packet may be sent before the partial chunk is
filled up. So resetting the checksum calculator makes the next packet to be sent with a wrong
checkusm.

> Block receiver unexpectedly throws an IOException complaining mismatched checksum
> ---------------------------------------------------------------------------------
>
>                 Key: HDFS-679
>                 URL: https://issues.apache.org/jira/browse/HDFS-679
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>
> When I run TestClientProtocolForPipelineRecovery, I always see that the block receiver
throws IOException complaining about mismatched checksum when receiving the last data packet.
It turns out the checksum of last packet was unexpectedly set to be zero.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message