hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John George (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2114) re-commission of a decommissioned node does not delete excess replica
Date Thu, 14 Jul 2011 21:55:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13065560#comment-13065560

John George commented on HDFS-2114:

> Matt Foley commented on HDFS-2114:
> ----------------------------------
> bq. The first use of {isNodeDown} is necessary because {downnode} could be 
> null in cases when we are checking for "Recommission".
> Yes, but in this case nodes[j].getName().equals(downnode) will return false 
> without any problem.

Also, I forgot to mention the check ensures that hasdown is not incremented. But, I agree
with you that the code does not look as good as it should. I have attached a modified patch
with some more changes.

> bq. My understanding is that this assert ensures that the current blk has only 
> ONE replica that is in decommissioned state.
> But it also seems to ensure that the only replica that is down is the LAST one 
> in the array of nodes for the block,
> due to the way this block works:
> {code}
>       int firstDecomNodeIndex = -1;
>       DatanodeInfo[] nodes = blk.getLocations();
>       for (int j = 0; j < nodes.length; j++) {     // for each replica of blk
>         ...
>         if (nodes[j].isDecommissioned()) {
>           if (firstDecomNodeIndex == -1) {
>             firstDecomNodeIndex = j;
>           }
>           continue;
>         }
>         assertEquals("Decom node is not at the end", firstDecomNodeIndex, -1);
>       }
> {code}
> Yet I don't see any code (eg in LocatedBlock.getLocations()) where downed 
> nodes are moved to the end of that list. Maybe somewhere else?

Yes I think it checks to ensure that the nodes are moved to the end of that list. The code
is in FSNamesystem.java. The last 5 lines in the following code is what does it.


   * Get block locations within the specified range.

   * @see ClientProtocol#getBlockLocations(String, long, long)


  LocatedBlocks getBlockLocations(String clientMachine, String src,

      long offset, long length) throws AccessControlException,

      FileNotFoundException, UnresolvedLinkException, IOException {

    LocatedBlocks blocks = getBlockLocations(src, offset, length, true, true);

    if (blocks != null) {

      //sort the blocks

      DatanodeDescriptor client = host2DataNodeMap.getDatanodeByHost(


      for (LocatedBlock b : blocks.getLocatedBlocks()) {

        clusterMap.pseudoSortByDistance(client, b.getLocations());

        // Move decommissioned datanodes to the bottom

        Arrays.sort(b.getLocations(), DFSUtil.DECOM_COMPARATOR);



    return blocks;




> re-commission of a decommissioned node does not delete excess replica
> ---------------------------------------------------------------------
>                 Key: HDFS-2114
>                 URL: https://issues.apache.org/jira/browse/HDFS-2114
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: John George
>            Assignee: John George
>         Attachments: HDFS-2114-2.patch, HDFS-2114-3.patch, HDFS-2114.patch
> If a decommissioned node is removed from the decommissioned list, namenode does not delete
the excess replicas it created while the node was decommissioned.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message