hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-679) Block receiver unexpectedly throws an IOException complaining mismatched checksum
Date Wed, 07 Oct 2009 19:01:32 GMT

    [ https://issues.apache.org/jira/browse/HDFS-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763192#action_12763192
] 

Hairong Kuang commented on HDFS-679:
------------------------------------

I'd like to propose a solution to the appending to a partial checksum chunk problem. The client
reads in the partial chunk into its buffer before any write and they will be part of the packets
sent to the datanode side before the chunk gets filled up. This solution maintains a good
property: any packet sent to datanodes starts at a checksum chunk boundary. This would not
increase complexity at the client side but would completely eliminate the handling of appending
to partial chunk at the datanode side. 

Another optimization to a block receiver at the datanode is not to overwrite block file if
a packet starts with data that the replica file already has. For the crc file, only the last
4 bytes are allowed to be overwritten.

> Block receiver unexpectedly throws an IOException complaining mismatched checksum
> ---------------------------------------------------------------------------------
>
>                 Key: HDFS-679
>                 URL: https://issues.apache.org/jira/browse/HDFS-679
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.21.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.21.0
>
>
> When I run TestClientProtocolForPipelineRecovery, I always see that the block receiver
throws IOException complaining about mismatched checksum when receiving the last data packet.
It turns out the checksum of last packet was unexpectedly set to be zero.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message