Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 72FF9D82C for ; Thu, 6 Sep 2012 01:24:08 +0000 (UTC) Received: (qmail 87422 invoked by uid 500); 6 Sep 2012 01:24:08 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 87374 invoked by uid 500); 6 Sep 2012 01:24:08 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 87173 invoked by uid 99); 6 Sep 2012 01:24:08 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Sep 2012 01:24:08 +0000 Date: Thu, 6 Sep 2012 12:24:08 +1100 (NCT) From: "Andy Isaacson (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: <135682698.42511.1346894648093.JavaMail.jiratomcat@arcas> In-Reply-To: <1445695877.36856.1345579838292.JavaMail.jiratomcat@arcas> Subject: [jira] [Updated] (HDFS-3828) Block Scanner rescans blocks too frequently MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Isaacson updated HDFS-3828: -------------------------------- Attachment: hdfs-3828-3.txt > Block Scanner rescans blocks too frequently > ------------------------------------------- > > Key: HDFS-3828 > URL: https://issues.apache.org/jira/browse/HDFS-3828 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 0.23.0, 2.0.0-alpha > Reporter: Andy Isaacson > Assignee: Andy Isaacson > Attachments: hdfs-3828-1.txt, hdfs-3828-2.txt, hdfs-3828-3.txt, hdfs3828.txt > > > {{BlockPoolSliceScanner#scan}} calls cleanUp every time it's invoked from {{DataBlockScanner#run}} via {{scanBlockPoolSlice}}. But cleanUp unconditionally roll()s the verificationLogs, so after two iterations we have lost the first iteration of block verification times. As a result a cluster with just one block repeatedly rescans it every 10 seconds: > {noformat} > 2012-08-16 15:59:57,884 INFO datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915 > 2012-08-16 16:00:07,904 INFO datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915 > 2012-08-16 16:00:17,925 INFO datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(391)) - Verification succeeded for BP-2101131164-172.29.122.91-1337906886255:blk_7919273167187535506_4915 > {noformat} > {quote} > To fix this, we need to avoid roll()ing the logs multiple times per period. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira