hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brahma Reddy Battula (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8586) Dead Datanode is allocated for write when client is from deadnode
Date Sat, 13 Jun 2015 06:05:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14584465#comment-14584465
] 

Brahma Reddy Battula commented on HDFS-8586:
--------------------------------------------

 *Test Code to reproduce this bug* 
{code}
public void testDeadDatanodeForBlockLocation() throws Exception {
    Configuration conf = new HdfsConfiguration();
    conf.setInt(DFSConfigKeys.DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY, 500);
    conf.setLong(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 1L);
    cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
    cluster.waitActive();
    
    String poolId = cluster.getNamesystem().getBlockPoolId();
    // wait for datanode to be marked live
    DataNode dn = cluster.getDataNodes().get(0);
    DatanodeRegistration reg = 
        DataNodeTestUtils.getDNRegistrationForBP(dn, poolId);
    
    DFSTestUtil.waitForDatanodeState(cluster, reg.getDatanodeUuid(), true, 20000);
    
    // Shutdown and wait for data node to be marked dead
    dn.shutdown();
    DFSTestUtil.waitForDatanodeState(cluster, reg.getDatanodeUuid(), false, 20000);
    System.out.println("Dn downXXX: " + dn.getDisplayName());
    
    Path file = new Path("afile");
    try (FSDataOutputStream outputStream = cluster.getFileSystem().create(file))
    {
      outputStream.writeChars("testContent");
    }
    
    
    BlockLocation block = cluster.getFileSystem().getFileBlockLocations(file, 0, 10)[0];
    System.out.println("Dn down: " + dn.getDisplayName());
    for(String node : block.getNames())
    {
      System.out.println(node);
      if(node.equals(dn.getDisplayName()))
          {
            fail("Not expecting the block in a dead node");
          }
    }
  }
{code}
 *Impact which I seen* 
{color:red}The cluster have 9 Datanode´╝înow stop 5. dfs.replications=3. Put files to HDFS
continuously, but some operations failed.{color} 


I think, Here we can deadnodes also... 

{code}
if (isGoodTarget(storage, blockSize, maxNodesPerRack, considerLoad,
        results, avoidStaleNodes, storageType)) {
      results.add(storage);
      // add node and related nodes to excludedNode
      return addToExcludedNodes(storage.getDatanodeDescriptor(), excludedNodes);
    }
{code}

> Dead Datanode is allocated for write when client is  from deadnode
> ------------------------------------------------------------------
>
>                 Key: HDFS-8586
>                 URL: https://issues.apache.org/jira/browse/HDFS-8586
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Brahma Reddy Battula
>            Assignee: Brahma Reddy Battula
>
>  *{color:blue}DataNode marked as Dead{color}* 
> 2015-06-11 19:39:00,862 | INFO  | org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@28ec166e
| BLOCK*  *removeDeadDatanode: lost heartbeat from XX.XX.39.33:25009*  | org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:584)
> 2015-06-11 19:39:00,863 | INFO  | org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager$Monitor@28ec166e
| Removing a node: /default/rack3/XX.XX.39.33:25009 | org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:488)
>   *{color:blue}Deadnode got Allocated{color}* 
> 2015-06-11 19:39:45,148 | WARN  | IPC Server handler 26 on 25000 | The cluster does not
contain node: /default/rack3/XX.XX.39.33:25009 | org.apache.hadoop.net.NetworkTopology.getDistance(NetworkTopology.java:616)
> 2015-06-11 19:39:45,149 | WARN  | IPC Server handler 26 on 25000 | The cluster does not
contain node: /default/rack3/XX.XX.39.33:25009 | org.apache.hadoop.net.NetworkTopology.getDistance(NetworkTopology.java:616)
> 2015-06-11 19:39:45,149 | WARN  | IPC Server handler 26 on 25000 | The cluster does not
contain node: /default/rack3/XX.XX.39.33:25009 | org.apache.hadoop.net.NetworkTopology.getDistance(NetworkTopology.java:616)
> 2015-06-11 19:39:45,149 | WARN  | IPC Server handler 26 on 25000 | The cluster does not
contain node: /default/rack3/XX.XX.39.33:25009 | org.apache.hadoop.net.NetworkTopology.getDistance(NetworkTopology.java:616)
> 2015-06-11 19:39:45,149 | INFO  | IPC Server handler 26 on 25000 | BLOCK*  *allocate
blk_1073754030_13252* {UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-e8d29773-dfc2-4224-b1d6-9b0588bca55e:NORMAL:{color:red}XX.XX.39.33:25009{color}|RBW],
 ReplicaUC[[DISK]DS-f7d2ab3c-88f7-470c-9097-84387c0bec83:NORMAL:XX.XX.38.32:25009|RBW], ReplicaUC[[DISK]DS-8c2a464a-ac81-4651-890a-dbfd07ddd95f:NORMAL:
*XX.XX.38.33:25009|RBW]]* } for /t1._COPYING_ | org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3657)
> 2015-06-11 19:39:45,191 | INFO  | IPC Server handler 35 on 25000 | BLOCK* allocate blk_1073754031_13253{UCState=UNDER_CONSTRUCTION,
truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ed8ad579-50c0-4e3e-8780-9776531763b6:NORMAL:XX.XX.39.31:25009|RBW],
ReplicaUC[[DISK]DS-19ddd6da-4a3e-481a-8445-dde5c90aaff3:NORMAL:XX.XX.37.32:25009|RBW], ReplicaUC[[DISK]DS-4ce4ce39-4973-42ce-8c7d-cb41f899db85:
{{NORMAL:XX.XX.37.33:25009}}   |RBW]]} for /t1._COPYING_ | org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveAllocatedBlock(FSNamesystem.java:3657)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message