From hdfs-issues-return-43927-apmail-hadoop-hdfs-issues-archive=hadoop.apache.org@hadoop.apache.org Mon Jul 2 11:53:24 2012 Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8750CD1EE for ; Mon, 2 Jul 2012 11:53:24 +0000 (UTC) Received: (qmail 39543 invoked by uid 500); 2 Jul 2012 11:53:24 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 39495 invoked by uid 500); 2 Jul 2012 11:53:24 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 39438 invoked by uid 99); 2 Jul 2012 11:53:23 -0000 Received: from issues-vm.apache.org (HELO issues-vm) (140.211.11.160) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2012 11:53:23 +0000 Received: from isssues-vm.apache.org (localhost [127.0.0.1]) by issues-vm (Postfix) with ESMTP id CA0AC14284F for ; Mon, 2 Jul 2012 11:53:22 +0000 (UTC) Date: Mon, 2 Jul 2012 11:53:22 +0000 (UTC) From: "amith (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: <1405841987.191.1341230002831.JavaMail.jiratomcat@issues-vm> In-Reply-To: <428496468.77343.1341210285210.JavaMail.jiratomcat@issues-vm> Subject: [jira] [Commented] (HDFS-3584) Blocks are getting marked as corrupt with append operation under high load. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-3584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13405025#comment-13405025 ] amith commented on HDFS-3584: ----------------------------- Here I can see client1 are not able to get the lease renewed with in soft limit period and client2 try to trigger lease recovery I can see this from the exception @ client Client log {noformat} 2012-06-28 19:40:56,452 INFO hdfs.TestHDFSAPI (TestHDFSAPI.java:writeFile(168)) - Writing File with client DFS[DFSClient[clientName=DFSClient_clientIDD36655341694508_139393306_1, ugi=B00902108 (auth:SIMPLE)]] File = /home/test/writefile_36673810701636 2012-06-28 19:41:04,619 INFO hdfs.TestHDFSAPI (TestHDFSAPI.java:writeFile(179)) - Creating File with client DFS[DFSClient[clientName=DFSClient_clientIDD36655341694508_139393306_1, ugi=B00902108 (auth:SIMPLE)]] File = /home/test/writefile_36673810701636 2012-06-28 19:42:28,004 INFO hdfs.TestHDFSAPI (TestHDFSAPI.java:writeFile(186)) - Closing File with client DFS[DFSClient[clientName=DFSClient_clientIDD36655341694508_139393306_1, ugi=B00902108 (auth:SIMPLE)]] File = /home/test/writefile_36673810701636 file size= 524288 2012-06-28 19:42:32,680 INFO hdfs.TestHDFSAPI (TestHDFSAPI.java:appendFile(76)) - Going Append File with client DFS[DFSClient[clientName=DFSClient_clientIDD36661058974201_139393306_1, ugi=B00902108 (auth:SIMPLE)]]File = /home/test/writefile_36673810701636file size= 524288 org.apache.hadoop.hdfs.protocol.RecoveryInProgressException: Failed to close file /home/test/writefile_36673810701636. Lease recovery is in progress. Try again later. {noformat} > Blocks are getting marked as corrupt with append operation under high load. > --------------------------------------------------------------------------- > > Key: HDFS-3584 > URL: https://issues.apache.org/jira/browse/HDFS-3584 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node > Affects Versions: 2.0.1-alpha > Reporter: Brahma Reddy Battula > > Scenario: > ========= > 1. There are 2 clients cli1 and cli2 cli1 write a file F1 and not closed > 2. The cli2 will call append on unclosed file and triggers a leaserecovery > 3. Cli1 is closed > 4. Lease recovery is completed and with updated GS in DN and got BlockReport since there is a mismatch in GS the block got corrupted > 5. Now we got a CommitBlockSync this will also fail since the File is already closed by cli1 and state in NN is Finalized -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira