hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8950) NameNode refresh doesn't remove DataNodes that are no longer in the allowed list
Date Tue, 27 Oct 2015 16:00:29 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14976612#comment-14976612
] 

Kihwal Lee commented on HDFS-8950:
----------------------------------

I tried to cherry-pick it, but {{TestDatanodeManager}} needs the following additional code.
{code:java}
  private static DatanodeManager mockDatanodeManager(
      FSNamesystem fsn, Configuration conf) throws IOException {
    BlockManager bm = Mockito.mock(BlockManager.class);
    DatanodeManager dm = new DatanodeManager(bm, fsn, conf);
    return dm;
  }
{code}

After this, {{TestDecommission}}, {{TestDatanodeManager}} and {{TestHostFileManager}} all
pass.

> NameNode refresh doesn't remove DataNodes that are no longer in the allowed list
> --------------------------------------------------------------------------------
>
>                 Key: HDFS-8950
>                 URL: https://issues.apache.org/jira/browse/HDFS-8950
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode, namenode
>    Affects Versions: 2.6.0
>            Reporter: Daniel Templeton
>            Assignee: Daniel Templeton
>              Labels: 2.7.2-candidate
>             Fix For: 2.8.0
>
>         Attachments: HDFS-8950.001.patch, HDFS-8950.002.patch, HDFS-8950.003.patch, HDFS-8950.004.patch,
HDFS-8950.005.patch
>
>
> If you remove a DN from NN's allowed host list (HDFS was HA) and then do NN refresh,
it doesn't remove it actually and the NN UI keeps showing that node. It may try to allocate
some blocks to that DN as well during an MR job.  This issue is independent from DN decommission.
> To reproduce:
> 1. Add a DN to dfs_hosts_allow
> 2. Refresh NN
> 3. Start DN. Now NN starts seeing DN.
> 4. Stop DN
> 5. Remove DN from dfs_hosts_allow
> 6. Refresh NN -> NN is still reporting DN as being used by HDFS.
> This is different from decom because there DN is added to exclude list in addition to
being removed from allowed list, and in that case everything works correctly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message