hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1264) 0.20: OOME in HDFS client made an unrecoverable HDFS block
Date Wed, 23 Jun 2010 20:24:52 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881859#action_12881859
] 

Todd Lipcon commented on HDFS-1264:
-----------------------------------

One of the issues here is that it's the *last* node in the pipeline that verifies the checksum.
So let's say I am writing to pipeline A,B,C. I first write a good packet (512) bytes, then
I write a packet with bad checksum. This gets written to disk on A and B without verifying
checksum. C verifies the checksum, gets the error, and forwards it back up the chain. The
client then drops C from the pipeline and initiates recovery. The recovery process gets the
replica info on A and B, both of which have length 1024 at this point. So it recovers the
block to a length that includes the bad checksum chunk, when we reopen the file, we have issues.

> 0.20: OOME in HDFS client made an unrecoverable HDFS block
> ----------------------------------------------------------
>
>                 Key: HDFS-1264
>                 URL: https://issues.apache.org/jira/browse/HDFS-1264
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node, hdfs client
>    Affects Versions: 0.20-append
>            Reporter: Todd Lipcon
>             Fix For: 0.20-append
>
>         Attachments: blk_logs_sorted.txt
>
>
> Ran into a bad issue in testing overnight. One of the writers experienced an OOME in
the middle of writing a checksum chunk to the stream inside a sync() call. It then proceeded
to retry recovery on each DN in the pipeline, but each recovery failed because its internal
checksum buffer was borked in some way - on the DNs I see "Unexpected checksum mismatch" errors
after each recovery attempt.
> When another client tried to recover the file using appendFile, it got the "Partial CRC
3766269197 does not match value computed the  last time file was closed" error (plus there
was only one replica left in targets). It thus failed to set up the append pipeline, and ran
into HDFS-1262.
> This was on 0.20-append, though it may happen on trunk as well.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message