From hdfs-issues-return-32093-apmail-hadoop-hdfs-issues-archive=hadoop.apache.org@hadoop.apache.org Thu Jan 12 18:40:04 2012 Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 57A1CB3BF for ; Thu, 12 Jan 2012 18:40:04 +0000 (UTC) Received: (qmail 33624 invoked by uid 500); 12 Jan 2012 18:40:04 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 33537 invoked by uid 500); 12 Jan 2012 18:40:03 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 33529 invoked by uid 99); 12 Jan 2012 18:40:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Jan 2012 18:40:03 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Jan 2012 18:40:00 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id B4333147A6F for ; Thu, 12 Jan 2012 18:39:39 +0000 (UTC) Date: Thu, 12 Jan 2012 18:39:39 +0000 (UTC) From: "Hari Mankude (Commented) (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: <598934640.35345.1326393579739.JavaMail.tomcat@hel.zones.apache.org> In-Reply-To: <1099139534.315.1325566341782.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Commented] (HDFS-2742) HA: observed dataloss in replication stress test MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HDFS-2742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185133#comment-13185133 ] Hari Mankude commented on HDFS-2742: ------------------------------------ This is a good idea. It keeps the log stream separate from the datanode based block msgs. We would need to handle "lost" msgs. For example, primary might have received a block finalized msg and the same msg was not delivered to standby. So, the block rbw msg on the standby queue will have to be purged after a timeout. > HA: observed dataloss in replication stress test > ------------------------------------------------ > > Key: HDFS-2742 > URL: https://issues.apache.org/jira/browse/HDFS-2742 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, ha, name-node > Affects Versions: HA branch (HDFS-1623) > Reporter: Todd Lipcon > Assignee: Todd Lipcon > Priority: Blocker > Attachments: hdfs-2742.txt, log-colorized.txt > > > The replication stress test case failed over the weekend since one of the replicas went missing. Still diagnosing the issue, but it seems like the chain of events was something like: > - a block report was generated on one of the nodes while the block was being written - thus the block report listed the block as RBW > - when the standby replayed this queued message, it was replayed after the file was marked complete. Thus it marked this replica as corrupt > - it asked the DN holding the corrupt replica to delete it. And, I think, removed it from the block map at this time. > - That DN then did another block report before receiving the deletion. This caused it to be re-added to the block map, since it was "FINALIZED" now. > - Replication was lowered on the file, and it counted the above replica as non-corrupt, and asked for the other replicas to be deleted. > - All replicas were lost. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira