hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei-Chiu Chuang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-11160) VolumeScanner reports write-in-progress replicas as corrupt incorrectly
Date Tue, 22 Nov 2016 04:35:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15685648#comment-15685648
] 

Wei-Chiu Chuang edited comment on HDFS-11160 at 11/22/16 4:35 AM:
------------------------------------------------------------------

Attach my simple fix in v001 patch.

In v001 fix, BlockSender constructor pre-loads last partial checksum from on-disk replica
if it is a finalized replica. This is simpler than adding a new field in FinalizedReplica
class and maintain the value of the field throughout the lifetime of the replica, at the cost
of potentially more disk access (because each BlockSender instantiation needs to reload checksum
again, regardless whether the replica is updated or not. In addition, the checksum is read
while holding FsDatasetImpl lock). I verified the unit test passed with this simple fix, and
fails without the fix.

Appreciate any comments!



was (Author: jojochuang):
Attach my simple fix in v001 patch.

In v001 fix, BlockSender constructor pre-loads last partial checksum from on-disk replica
if it is a finalized replica. This is simpler than adding a new field in FinalizedReplica
class and maintain the value of the field throughout the lifetime of the replica, at the cost
of potentially more disk access (because each BlockSender instantiation needs to reload checksum
again, regardless whether the replica is updated or not). I verified the unit test passed
with this simple fix, and fails without the fix.

Appreciate any comments!


> VolumeScanner reports write-in-progress replicas as corrupt incorrectly
> -----------------------------------------------------------------------
>
>                 Key: HDFS-11160
>                 URL: https://issues.apache.org/jira/browse/HDFS-11160
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>         Environment: CDH5.7.4
>            Reporter: Wei-Chiu Chuang
>            Assignee: Wei-Chiu Chuang
>         Attachments: HDFS-11160.001.patch, HDFS-11160.reproduce.patch
>
>
> Due to a race condition initially reported in HDFS-6804, VolumeScanner may erroneously
detect good replicas as corrupt. This is serious because in some cases it results in data
loss if all replicas are declared corrupt.
> We are investigating an incidence that caused very high block corruption rate in a relatively
small cluster. Initially, we thought HDFS-11056 is to blame. However, after applying HDFS-11056,
we are still seeing VolumeScanner reporting corrupt replicas.
> It turns out that if a replica is being appended while VolumeScanner is scanning it,
VolumeScanner may use the new checksum to compare against old data, causing checksum mismatch.
> I have a unit test to reproduce the error. Will attach later.
> To fix it, I propose a FinalizedReplica object should also have a lastChecksum field
like ReplicaBeingWritten, and BlockSender should use the in-memory lastChecksum to verify
the partial data in the last chunk on disk. File this jira to discuss a good fix for this
issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message