hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7430) Refactor the BlockScanner to use O(1) memory and use multiple threads
Date Mon, 24 Nov 2014 22:54:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223709#comment-14223709

Colin Patrick McCabe commented on HDFS-7430:

* Fix findbugs warning

* Fix javac warning

Other stuff looks like an slf4j problem: getting a bunch of stuff like this:
 Time elapsed: 0 sec  <<< ERROR!
java.lang.ClassCastException: org.slf4j.impl.Log4jLoggerAdapter cannot be cast to org.apache.commons.logging.impl.Log4JLogger
    at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotTestHelper.setLevel2OFF(SnapshotTestHelper.java:105)
    at org.apache.hadoop.hdfs.server.namenode.snapshot.SnapshotTestHelper.disableLogs(SnapshotTestHelper.java:92)
    at org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots.<init>(TestRenameWithSnapshots.java:80)

Since I changed the scanner from using log4j to using slf4j

> Refactor the BlockScanner to use O(1) memory and use multiple threads
> ---------------------------------------------------------------------
>                 Key: HDFS-7430
>                 URL: https://issues.apache.org/jira/browse/HDFS-7430
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.7.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-7430.002.patch, HDFS-7430.003.patch, memory.png
> We should update the BlockScanner to use a constant amount of memory by keeping track
of what block was scanned last, rather than by tracking the scan status of all blocks in memory.
 Also, instead of having just one thread, we should have a verification thread per hard disk
(or other volume), scanning at a configurable rate of bytes per second.

This message was sent by Atlassian JIRA

View raw message