Return-Path: Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org Received: (qmail 88277 invoked from network); 5 Dec 2006 14:07:43 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 5 Dec 2006 14:07:43 -0000 Received: (qmail 59568 invoked by uid 500); 5 Dec 2006 14:07:51 -0000 Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org Received: (qmail 59539 invoked by uid 500); 5 Dec 2006 14:07:51 -0000 Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-dev@lucene.apache.org Received: (qmail 59530 invoked by uid 99); 5 Dec 2006 14:07:51 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Dec 2006 06:07:51 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO brutus.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Dec 2006 06:07:42 -0800 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 4C9A47142D7 for ; Tue, 5 Dec 2006 06:07:22 -0800 (PST) Message-ID: <6343156.1165327642311.JavaMail.jira@brutus> Date: Tue, 5 Dec 2006 06:07:22 -0800 (PST) From: "dhruba borthakur (JIRA)" To: hadoop-dev@lucene.apache.org Subject: [jira] Updated: (HADOOP-774) Datanodes fails to heartbeat when a directory with a large number of blocks is deleted In-Reply-To: <10916275.1165019421088.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ http://issues.apache.org/jira/browse/HADOOP-774?page=all ] dhruba borthakur updated HADOOP-774: ------------------------------------ Attachment: chunkinvalidateBlocks2.java Incorporated review comments. > Datanodes fails to heartbeat when a directory with a large number of blocks is deleted > -------------------------------------------------------------------------------------- > > Key: HADOOP-774 > URL: http://issues.apache.org/jira/browse/HADOOP-774 > Project: Hadoop > Issue Type: Bug > Components: dfs > Reporter: dhruba borthakur > Assigned To: dhruba borthakur > Attachments: chunkinvalidateBlocks2.java > > > If a user removes a few files that are huge, it causes the namenode to send BlockInvalidate command to the relevant Datanodes. The Datanode process the blockInvalidate command as part of its heartbeat thread. If the number of blocks to be invalidated is huge, the datanode takes a long time to process it. This causes the datanode to not send new heartbeats to the namenode. The namenode declares the datanode as dead! > 1. One option is to process the blockInvalidate as a separate thread from the heartbeat thread in the Datanode. > 2. Another option would be to constrain the namenode to send a max (e.g. 500) blocks per blockInvalidate message. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira