hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jitendra Nath Pandey (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1655) Federation: DatablockScanner should scan blocks for all the block pools.
Date Tue, 01 Mar 2011 19:07:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001026#comment-13001026
] 

Jitendra Nath Pandey commented on HDFS-1655:
--------------------------------------------

Changes in the patch.
- The block pools are scanned one at a time. Most of the original DataBlockScanner functionality
is moved to BlockPoolScanner. The DataBlockScanner runs a scanner thread as before and selects
next block pool to scan and invokes BlockPoolScanner to do the scan work. 
- One BlockPoolScanner is instantiated for each block pool which maintains the state of the
scan, blockMap and blockInfoSet similar to what DataBlockScanner had before.
- The current verification log file is created for each block pool and once the block pool
scan is complete it is rolled into the previous log file, before processing the next block
pool. Therefore, at any instant there should be only one current verification log file. 
- To find the next block pool to scan, the datablock scanner iterates over the set of blockPools.
However, if a datanode is restarted it needs to pick the block pool which was being scanned
before. Therefore, first it looks for the block pool that contains the current verification
log file and that is used as the starting block pool id. If no current files are found it
starts with first block-pool.

> Federation: DatablockScanner should scan blocks for all the block pools.
> ------------------------------------------------------------------------
>
>                 Key: HDFS-1655
>                 URL: https://issues.apache.org/jira/browse/HDFS-1655
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>    Affects Versions: Federation Branch
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>             Fix For: Federation Branch
>
>         Attachments: HDFS-1655.1.patch
>
>
> DataBlockScanner changes are needed to work with federation.
> Goal is to have DataBlockScanner visit one volume at a time, scanning block pools under
it one at a time.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message