hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7994) Detect if resevered EC Block ID is already used
Date Fri, 10 Apr 2015 00:40:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14488639#comment-14488639
] 

Tsz Wo Nicholas Sze commented on HDFS-7994:
-------------------------------------------

You are right that the current implementation already has this function but it does an additional
lookup.  Rewrite getStoredBlock(..) as below.
{code}
public BlockInfo getStoredBlock(Block block) {
    if (BlockIdManager.isStripedBlockID(block.getBlockId())) {
      BlockInfo info =  blocksMap.getStoredBlock(
          new Block(BlockIdManager.convertToStripedID(block.getBlockId())));     //AAA
      if (info == null) {
        info = blocksMap.getStoredBlock(block);              // BBB1
      }
      return info;
    } else {
      return blocksMap.getStoredBlock(block);              // BBB2
    }
  }
{code}
When the input block ID is a striped block ID and not found at AAA, it tries BBB1.  We can
eliminate BBB1 if we know that there is no non-EC block using a striped block ID.

This JIRA is to
# check whether there is a non-EC block using a striped block ID during fsimage/edit loading.
# If yes, use the current logic.  Otherwise, skip BBB1.  i.e.
{code}
public BlockInfo getStoredBlock(Block block) {
    if (BlockIdManager.isStripedBlockID(block.getBlockId())) {
      BlockInfo info =  blocksMap.getStoredBlock(
          new Block(BlockIdManager.convertToStripedID(block.getBlockId())));     //AAA
      if (info == null && hasNonEcBlockUsingStripID) {
        info = blocksMap.getStoredBlock(block);              // BBB1
      }
      return info;
    } else {
      return blocksMap.getStoredBlock(block);              // BBB2
    }
  }
{code}


> Detect if resevered EC Block ID is already used
> -----------------------------------------------
>
>                 Key: HDFS-7994
>                 URL: https://issues.apache.org/jira/browse/HDFS-7994
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Hui Zheng
>
> Since random block IDs were supported by some early version of HDFS, the block ID reserved
for EC blocks could be already used by some existing blocks in a cluster. During NameNode
startup, it detects if there are reserved EC block IDs used by non-EC blocks. If it is the
case, NameNode will do an additional blocksMap lookup when there is a miss in a blockGroupsMap
lookup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message