Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6BA3C10D9F for ; Thu, 30 Jan 2014 21:38:30 +0000 (UTC) Received: (qmail 4365 invoked by uid 500); 30 Jan 2014 21:38:12 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 4314 invoked by uid 500); 30 Jan 2014 21:38:11 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 4303 invoked by uid 99); 30 Jan 2014 21:38:11 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 30 Jan 2014 21:38:11 +0000 Date: Thu, 30 Jan 2014 21:38:11 +0000 (UTC) From: "Kihwal Lee (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Resolved] (HDFS-4183) Throttle block recovery MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-4183. ------------------------------ Resolution: Duplicate Target Version/s: 2.1.0-beta, 3.0.0 (was: 3.0.0, 2.1.0-beta) > Throttle block recovery > ----------------------- > > Key: HDFS-4183 > URL: https://issues.apache.org/jira/browse/HDFS-4183 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode > Affects Versions: 0.23.4, 2.0.2-alpha > Reporter: Kihwal Lee > Assignee: Kihwal Lee > Priority: Critical > > When a large number of files are abandoned without closing, a storm of lease expiration follows in about an hour (lease hard limit). For the last block of each file, block recovery is initiated and when the datanode is done, it calls commitBlockSynchronization() is called against namenode. A burst of these calls can slow down namenode considerably. We need to throttle block recovery and/or speed up the rate at which commitBlockSynchronization() is served. -- This message was sent by Atlassian JIRA (v6.1.5#6160)