hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John George (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2114) re-commission of a decommissioned node does not delete excess replica
Date Thu, 14 Jul 2011 21:55:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065560#comment-13065560
] 

John George commented on HDFS-2114:
-----------------------------------


> Matt Foley commented on HDFS-2114:
> ----------------------------------
> 
> bq. The first use of {isNodeDown} is necessary because {downnode} could be 
> null in cases when we are checking for "Recommission".
> 
> Yes, but in this case nodes[j].getName().equals(downnode) will return false 
> without any problem.

Also, I forgot to mention the check ensures that hasdown is not incremented. But, I agree
with you that the code does not look as good as it should. I have attached a modified patch
with some more changes.

> 
> bq. My understanding is that this assert ensures that the current blk has only 
> ONE replica that is in decommissioned state.
> 
> But it also seems to ensure that the only replica that is down is the LAST one 
> in the array of nodes for the block,
> due to the way this block works:
> {code}
>       int firstDecomNodeIndex = -1;
>       DatanodeInfo[] nodes = blk.getLocations();
>       for (int j = 0; j < nodes.length; j++) {     // for each replica of blk
>         ...
>         if (nodes[j].isDecommissioned()) {
>           if (firstDecomNodeIndex == -1) {
>             firstDecomNodeIndex = j;
>           }
>           continue;
>         }
>         assertEquals("Decom node is not at the end", firstDecomNodeIndex, -1);
>       }
> {code}
> Yet I don't see any code (eg in LocatedBlock.getLocations()) where downed 
> nodes are moved to the end of that list. Maybe somewhere else?

Yes I think it checks to ensure that the nodes are moved to the end of that list. The code
is in FSNamesystem.java. The last 5 lines in the following code is what does it.

{code}
  /**

   * Get block locations within the specified range.

   * @see ClientProtocol#getBlockLocations(String, long, long)

   */

  LocatedBlocks getBlockLocations(String clientMachine, String src,

      long offset, long length) throws AccessControlException,

      FileNotFoundException, UnresolvedLinkException, IOException {

    LocatedBlocks blocks = getBlockLocations(src, offset, length, true, true);

    if (blocks != null) {

      //sort the blocks

      DatanodeDescriptor client = host2DataNodeMap.getDatanodeByHost(

          clientMachine);

      for (LocatedBlock b : blocks.getLocatedBlocks()) {

        clusterMap.pseudoSortByDistance(client, b.getLocations());



        // Move decommissioned datanodes to the bottom

        Arrays.sort(b.getLocations(), DFSUtil.DECOM_COMPARATOR);

      }

    }

    return blocks;

  }

{code}

> 


> re-commission of a decommissioned node does not delete excess replica
> ---------------------------------------------------------------------
>
>                 Key: HDFS-2114
>                 URL: https://issues.apache.org/jira/browse/HDFS-2114
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: John George
>            Assignee: John George
>         Attachments: HDFS-2114-2.patch, HDFS-2114-3.patch, HDFS-2114.patch
>
>
> If a decommissioned node is removed from the decommissioned list, namenode does not delete
the excess replicas it created while the node was decommissioned.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message