hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongjun Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7403) Inaccurate javadoc of BlockUCState#COMPLETE state
Date Thu, 20 Nov 2014 04:30:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218995#comment-14218995

Yongjun Zhang commented on HDFS-7403:

I noticed the following code in BlockManager.java, which means even if a block doesn't have
enough replicas, it can still be marked as complete:

   * Force the given block in the given file to be marked as complete,
   * regardless of whether enough replicas are present. This is necessary
   * when tailing edit logs as a Standby.
  public BlockInfo forceCompleteBlock(final BlockCollection bc,
      final BlockInfoUnderConstruction block) throws IOException {
    return completeBlock(bc, block, true);
that calls 
   * Convert a specified block of the file to a complete block.
   * @param bc file
   * @param blkIndex  block index in the file
   * @throws IOException if the block does not have at least a minimal number
   * of replicas reported from data-nodes.
  private BlockInfo completeBlock(final BlockCollection bc,
      final int blkIndex, boolean force) throws IOException {
    if(blkIndex < 0)
      return null;
    BlockInfo curBlock = bc.getBlocks()[blkIndex];
      return curBlock;
    BlockInfoUnderConstruction ucBlock = (BlockInfoUnderConstruction)curBlock;
    int numNodes = ucBlock.numNodes();
    if (!force && numNodes < minReplication)
      throw new IOException("Cannot complete block: " +
          "block does not satisfy minimal replication requirement.");
    if(!force && ucBlock.getBlockUCState() != BlockUCState.COMMITTED)
      throw new IOException(
          "Cannot complete block: block has not been COMMITTED by the client");
    BlockInfo completeBlock = ucBlock.convertToCompleteBlock();
    // replace penultimate block in file
    bc.setBlock(blkIndex, completeBlock);
    // Since safe-mode only counts complete blocks, and we now have
    // one more complete block, we need to adjust the total up, and
    // also count it as safe, if we have at least the minimum replica
    // count. (We may not have the minimum replica count yet if this is
    // a "forced" completion when a file is getting closed by an
    // OP_CLOSE edit on the standby).
    namesystem.adjustSafeModeBlockTotals(0, 1);
        Math.min(numNodes, minReplication));
    // replace block in the blocksMap
    return blocksMap.replaceBlock(completeBlock);

Well, the javadoc of COMPLETE says it means "The block has at least one", thus it is still
confusing. I was thinking about changing the COMPLETE javadoc to be something like
In common case, COMPLETE means that the block need to have >= minimal replication replicas"
(link to minimalReplica def). However, a block may be forced to COMPLETE state even if it
doesn't have any replica. (link to the above method)

> Inaccurate javadoc of  BlockUCState#COMPLETE state
> --------------------------------------------------
>                 Key: HDFS-7403
>                 URL: https://issues.apache.org/jira/browse/HDFS-7403
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>            Priority: Trivial
>         Attachments: HDFS-7403.001.patch
> The current javadoc says 
> {code}
>  /**
>    * States, which a block can go through while it is under construction.
>    */
>   static public enum BlockUCState {
>     /**
>      * Block construction completed.<br>
>      * The block has at least one {@link ReplicaState#FINALIZED} replica,
>      * and is not going to be modified.
>      */
> {code}
> However, COMPLETE blocks mean those that have reached minimal replication  "dfs.namenode.replication.min",
which could be different than one.
> Creating this jira to fix the javadoc.

This message was sent by Atlassian JIRA

View raw message