Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 20 Nov 2014 04:30:34 +0000 (UTC)
From: "Yongjun Zhang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12755676.1416153847000.541745.1416457834437@Atlassian.JIRA>
In-Reply-To: <JIRA.12755676.1416153847000@Atlassian.JIRA>
References: <JIRA.12755676.1416153847000@Atlassian.JIRA>
 <JIRA.12755676.1416153847992@arcas>
Subject: [jira] [Commented] (HDFS-7403) Inaccurate javadoc of
  BlockUCState#COMPLETE state
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218995#comment-14218995 ] 

Yongjun Zhang commented on HDFS-7403:
-------------------------------------

I noticed the following code in BlockManager.java, which means even if a block doesn't have enough replicas, it can still be marked as complete:

 /**
   * Force the given block in the given file to be marked as complete,
   * regardless of whether enough replicas are present. This is necessary
   * when tailing edit logs as a Standby.
   */
  public BlockInfo forceCompleteBlock(final BlockCollection bc,
      final BlockInfoUnderConstruction block) throws IOException {
    block.commitBlock(block);
    return completeBlock(bc, block, true);
  }
that calls 
{code}
 /**
   * Convert a specified block of the file to a complete block.
   * @param bc file
   * @param blkIndex  block index in the file
   * @throws IOException if the block does not have at least a minimal number
   * of replicas reported from data-nodes.
   */
  private BlockInfo completeBlock(final BlockCollection bc,
      final int blkIndex, boolean force) throws IOException {
    if(blkIndex < 0)
      return null;
    BlockInfo curBlock = bc.getBlocks()[blkIndex];
    if(curBlock.isComplete())
      return curBlock;
    BlockInfoUnderConstruction ucBlock = (BlockInfoUnderConstruction)curBlock;
    int numNodes = ucBlock.numNodes();
    if (!force && numNodes < minReplication)
      throw new IOException("Cannot complete block: " +
          "block does not satisfy minimal replication requirement.");
    if(!force && ucBlock.getBlockUCState() != BlockUCState.COMMITTED)
      throw new IOException(
          "Cannot complete block: block has not been COMMITTED by the client");
    BlockInfo completeBlock = ucBlock.convertToCompleteBlock();
    // replace penultimate block in file
    bc.setBlock(blkIndex, completeBlock);
    
    // Since safe-mode only counts complete blocks, and we now have
    // one more complete block, we need to adjust the total up, and
    // also count it as safe, if we have at least the minimum replica
    // count. (We may not have the minimum replica count yet if this is
    // a "forced" completion when a file is getting closed by an
    // OP_CLOSE edit on the standby).
    namesystem.adjustSafeModeBlockTotals(0, 1);
    namesystem.incrementSafeBlockCount(
        Math.min(numNodes, minReplication));
    
    // replace block in the blocksMap
    return blocksMap.replaceBlock(completeBlock);
  }
{code}

Well, the javadoc of COMPLETE says it means "The block has at least one", thus it is still confusing. I was thinking about changing the COMPLETE javadoc to be something like
{code}
In common case, COMPLETE means that the block need to have >= minimal replication replicas" (link to minimalReplica def). However, a block may be forced to COMPLETE state even if it doesn't have any replica. (link to the above method)
{code}


> Inaccurate javadoc of  BlockUCState#COMPLETE state
> --------------------------------------------------
>
>                 Key: HDFS-7403
>                 URL: https://issues.apache.org/jira/browse/HDFS-7403
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>            Priority: Trivial
>         Attachments: HDFS-7403.001.patch
>
>
> The current javadoc says 
> {code}
>  /**
>    * States, which a block can go through while it is under construction.
>    */
>   static public enum BlockUCState {
>     /**
>      * Block construction completed.<br>
>      * The block has at least one {@link ReplicaState#FINALIZED} replica,
>      * and is not going to be modified.
>      */
>     COMPLETE,
> {code}
> However, COMPLETE blocks mean those that have reached minimal replication  "dfs.namenode.replication.min", which could be different than one.
>  
> Creating this jira to fix the javadoc.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)