Return-Path: 
 <hdfs-issues-return-102840-apmail-hadoop-hdfs-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 7397A104DF
	for <apmail-hadoop-hdfs-issues-archive@minotaur.apache.org>;
 Mon, 24 Nov 2014 22:54:13 +0000 (UTC)
Received: (qmail 99995 invoked by uid 500); 24 Nov 2014 22:54:13 -0000
Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org
Received: (qmail 99942 invoked by uid 500); 24 Nov 2014 22:54:13 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Reply-To: hdfs-issues@hadoop.apache.org
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 99931 invoked by uid 99); 24 Nov 2014 22:54:13 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Nov 2014 22:54:13 +0000
Date: Mon, 24 Nov 2014 22:54:13 +0000 (UTC)
From: "Colin Patrick McCabe (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12757328.1416803753000.10778.1416869653127@Atlassian.JIRA>
In-Reply-To: <JIRA.12757328.1416803753000@Atlassian.JIRA>
References: <JIRA.12757328.1416803753000@Atlassian.JIRA>
 <JIRA.12757328.1416803753163@arcas>
Subject: [jira] [Commented] (HDFS-7430) Refactor the BlockScanner to use
 O(1) memory and use multiple threads
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223709#comment-14223709 ] 

Colin Patrick McCabe commented on HDFS-7430:
--------------------------------------------

* Fix findbugs warning

* Fix javac warning

Other stuff looks like an slf4j problem: getting a bunch of stuff like this:
{code}
testRenameDirAndDeleteSnapshot_2(org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots)  Time elapsed: 0 sec  <<< ERROR!
java.lang.ClassCastException: org.slf4j.impl.Log4jLoggerAdapter cannot be cast to org.apache.commons.logging.impl.Log4JLogger
    at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotTestHelper.setLevel2OFF(SnapshotTestHelper.java:105)
    at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotTestHelper.disableLogs(SnapshotTestHelper.java:92)
    at org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots.<init>(TestRenameWithSnapshots.java:80)
{code}

Since I changed the scanner from using log4j to using slf4j

> Refactor the BlockScanner to use O(1) memory and use multiple threads
> ---------------------------------------------------------------------
>
>                 Key: HDFS-7430
>                 URL: https://issues.apache.org/jira/browse/HDFS-7430
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.7.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-7430.002.patch, HDFS-7430.003.patch, memory.png
>
>
> We should update the BlockScanner to use a constant amount of memory by keeping track of what block was scanned last, rather than by tracking the scan status of all blocks in memory.  Also, instead of having just one thread, we should have a verification thread per hard disk (or other volume), scanning at a configurable rate of bytes per second.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)