hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Foley (JIRA)" <j...@apache.org>
Subject [jira] Created: (HDFS-1687) HDFS Federation: DirectoryScanner changes for federation
Date Tue, 01 Mar 2011 23:10:36 GMT
HDFS Federation: DirectoryScanner changes for federation

                 Key: HDFS-1687
                 URL: https://issues.apache.org/jira/browse/HDFS-1687
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: data-node
    Affects Versions: Federation Branch
            Reporter: Matt Foley
            Assignee: Matt Foley
             Fix For: Federation Branch

DirectoryScanner scans substantially all of the directory tree of entire volumes.  It needs
to be extended to work with Blockpools in Federation.  

Design notes:

1. The subdirectories of active bpid's will be scanned.  Active bpid's are those associated
with currently connected Namenodes.  Each Volume knows the set of all active bpid's, via volume.map.keySet().
 I'll add a package-private accessor in FSVolume to return the set of active bpid's for use
by DirectoryScanner, DataBlockScanner, etc.  DirectoryScanner will ignore inactive bpid's
subdirectories; see item below.  

2. There is no need to compare the volume set of active bpid's with the global set, because
the way the code works, they really can't be different.  If differences arise, they will be
automatically fixed by the next restart of either the Datanode or the Namenode.

3. Inactive bpid's will be ignored.  Until we are connected to the owner Namenode, we cannot
know whether a bpid subdirectory is correctly formatted, has snapshot data, etc.  So it doesn't
make sense to try to manage the data under an inactive bpid.

4. DirectoryScanner is currently instantiated and periodically triggered by DataBlockScanner.
 Other than both being "scanners", these two modules have little in common, and the triggering
code is confusing.  (DirectoryScanner scans filesystem directory trees every hour, to detect
and fix inconsistencies between disk directories and ReplicasMap.  DataBlockScanner runs every
3 weeks, and traverses all block files, actually reading them out and checksumming them to
detect block corruption.)

Separating them, and running DirectoryScanner under its own periodic scheduler, is a small
change that will make the code much clearer.  It already runs on its own FixedThreadPool Executor,
so it is easy to change it to a ScheduledThreadPool, and instantiate it from DataNode.postStartInit()
at the same time as initBlockScanner() is called.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message