hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9220) Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum
Date Mon, 04 Jan 2016 14:33:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15081210#comment-15081210
] 

Kihwal Lee commented on HDFS-9220:
----------------------------------

This one goes with HDFS-4660 and HDFS-8722.  HDFS-4660 fixed a data corruption issue and HDFS-8722
fixed a performance regression in HDFS-4660. These changes introduced this bug (HDFS-9220).
 So, if you want to fix the corruption issue, you need to pull in all three. The issue fixed
in HDFS-4660 was originally seen very rarely, but after the datanode layout change, it was
occurring much more frequently along with other bugs we fixed regarding pipeline failures.
 If you plan to pull in the layout fix in the next release of 2.6,  HDFS-4660 may not be as
critical.

> Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum
> ----------------------------------------------------------------------------------------
>
>                 Key: HDFS-9220
>                 URL: https://issues.apache.org/jira/browse/HDFS-9220
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.7.1
>            Reporter: Bogdan Raducanu
>            Assignee: Jing Zhao
>            Priority: Blocker
>             Fix For: 3.0.0, 2.7.2
>
>         Attachments: HDFS-9220.000.patch, HDFS-9220.001.patch, HDFS-9220.002.patch, test2.java
>
>
> Exception:
> 2015-10-09 14:59:40 WARN  DFSClient:1150 - fetchBlockByteRange(). Got a checksum exception
for /tmp/file0.05355529331575182 at BP-353681639-10.10.10.10-1437493596883:blk_1075692769_9244882:0
from DatanodeInfoWithStorage[10.10.10.10]:5001
> All 3 replicas cause this exception and the read fails entirely with:
> BlockMissingException: Could not obtain block: BP-353681639-10.10.10.10-1437493596883:blk_1075692769_9244882
file=/tmp/file0.05355529331575182
> Code to reproduce is attached.
> Does not happen in 2.7.0.
> Data is read correctly if checksum verification is disabled.
> More generally, the failure happens when reading from the last block of a file and the
last block has <= 512 bytes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message