Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Message-ID: <21174896.14271280193019079.JavaMail.jira@thor>
Date: Mon, 26 Jul 2010 21:10:19 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Subject: [jira] Commented: (HDFS-1227) UpdateBlock fails due to unmatched
 file length
In-Reply-To: <15791497.47981276748303883.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892588#action_12892588 ] 

Todd Lipcon commented on HDFS-1227:
-----------------------------------

Hi Thanh. Since the writer is interrupted before getting the recovery info, it will return the new length, with the effect of the write, like you said. But when the synchronized length is calculated, it's taken as the minimum of the lengths (ie the length at DN1), which is what we want.

Did you test this against the append branch with HDFS-1186 applied on top? (Note that 1186 has not yet been committed to the branch)

> UpdateBlock fails due to unmatched file length
> ----------------------------------------------
>
>                 Key: HDFS-1227
>                 URL: https://issues.apache.org/jira/browse/HDFS-1227
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 0.20-append
>            Reporter: Thanh Do
>
> - Summary: client append is not atomic, hence, it is possible that
> when retrying during append, there is an exception in updateBlock
> indicating unmatched file length, making append failed.
>  
> - Setup:
> + # available datanodes = 3
> + # disks / datanode = 1
> + # failures = 1
> + failure type = bad disk
> + When/where failure happens = (see below)
> + This bug is non-deterministic, to reproduce it, add a sufficient sleep before out.write() in BlockReceiver.receivePacket() in dn1 and dn2 but not dn3
>  
> - Details:
>  Suppose client appends 16 bytes to block X which has length 16 bytes at dn1, dn2, dn3.
> Dn1 is primary. The pipeline is dn3-dn2-dn1. recoverBlock succeeds.
> Client starts sending data to the dn3 - the first datanode in pipeline.
> dn3 forwards the packet to downstream datanodes, and starts writing
> data to its disk. Suppose there is an exception in dn3 when writing to disk.
> Client gets the exception, it starts the recovery code by calling dn1.recoverBlock() again.
> dn1 in turn calls dn2.getMetadataInfo() and dn1.getMetaDataInfo() to build the syncList.
> Suppose at the time getMetadataInfo() is called at both datanodes (dn1 and dn2),
> the previous packet (which is sent from dn3) has not come to disk yet.
> Hence, the block Info given by getMetaDataInfo contains the length of 16 bytes.
> But after that, the packet "comes" to disk, making the block file length now becomes 32 bytes.
> Using the syncList (with contains block info with length 16 byte), dn1 calls updateBlock at
> dn2 and dn1, which will failed, because the length of new block info (given by updateBlock,
> which is 16 byte) does not match with its actual length on disk (which is 32 byte)
>  
> Note that this bug is non-deterministic. Its depends on the thread interleaving
> at datanodes.
> This bug was found by our Failure Testing Service framework:
> http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-98.html
> For questions, please email us: Thanh Do (thanhdo@cs.wisc.edu) and 
> Haryadi Gunawi (haryadi@eecs.berkeley.edu)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.