hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-7596) NameNode should prune dead storages from storageMap
Date Sat, 10 Jan 2015 01:55:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272239#comment-14272239
] 

Arpit Agarwal edited comment on HDFS-7596 at 1/10/15 1:54 AM:
--------------------------------------------------------------

Thanks for the review [~lars_francke].

The .002 patch fixes failed test case {{TestBlockManager#testSafeModeIBR}} and addresses part
of Lars' feedback. The TestBalancer failure is due to HDFS-7267.

{code}
      excessStorages = new HashMap<String, DatanodeStorageInfo>(storageMap);
{code}
This is deliberate to simplify back-porting the patch if needed (branch-2.6 is required to
support Java 6).

Differing chunks between the two patches for quick reference.

{code}
> @@ -416,6 +416,46 @@ public void updateHeartbeatState(StorageReport[] reports, long cacheCapacity,
33,36c33
< +        final String storageId = report.getStorage().getStorageID();
< +        if (excessStorages.get(storageId) != null) {
< +          excessStorages.remove(storageId);
< +        }
---
> +        excessStorages.remove(report.getStorage().getStorageID());

And

> @@ -571,11 +571,13 @@ public void testSafeModeIBR() throws Exception {
>      reset(node);
>      bm.getDatanodeManager().registerDatanode(nodeReg);
>      verify(node).updateRegInfo(nodeReg);
> -    assertEquals(0, ds.getBlockReportCount()); // ready for report again
>      // send block report, should be processed after restart
>      reset(node);
>      bm.processReport(node, new DatanodeStorage(ds.getStorageID()),
> -        new BlockListAsLongs(null, null));
> +                     new BlockListAsLongs(null, null));
> +    // Reinitialize as registration with empty storage list pruned
> +    // node.storageMap.
> +    ds = node.getStorageInfos()[0];
{code}


was (Author: arpitagarwal):
Thanks for the review [~lars_francke].

The .002 patch fixes failed test case {{TestBlockManager#testSafeModeIBR}} and addresses part
of Lars' feedback.

{code}
      excessStorages = new HashMap<String, DatanodeStorageInfo>(storageMap);
{code}
This is deliberate to simplify back-porting the patch if needed (branch-2.6 is required to
support Java 6).

Differing chunks between the two patches for quick reference.

{code}
> @@ -416,6 +416,46 @@ public void updateHeartbeatState(StorageReport[] reports, long cacheCapacity,
33,36c33
< +        final String storageId = report.getStorage().getStorageID();
< +        if (excessStorages.get(storageId) != null) {
< +          excessStorages.remove(storageId);
< +        }
---
> +        excessStorages.remove(report.getStorage().getStorageID());

And

> @@ -571,11 +571,13 @@ public void testSafeModeIBR() throws Exception {
>      reset(node);
>      bm.getDatanodeManager().registerDatanode(nodeReg);
>      verify(node).updateRegInfo(nodeReg);
> -    assertEquals(0, ds.getBlockReportCount()); // ready for report again
>      // send block report, should be processed after restart
>      reset(node);
>      bm.processReport(node, new DatanodeStorage(ds.getStorageID()),
> -        new BlockListAsLongs(null, null));
> +                     new BlockListAsLongs(null, null));
> +    // Reinitialize as registration with empty storage list pruned
> +    // node.storageMap.
> +    ds = node.getStorageInfos()[0];
{code}

> NameNode should prune dead storages from storageMap
> ---------------------------------------------------
>
>                 Key: HDFS-7596
>                 URL: https://issues.apache.org/jira/browse/HDFS-7596
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.6.0
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>         Attachments: HDFS-7596.01.patch, HDFS-7596.02.patch
>
>
> The NameNode must be able to prune storages that are no longer reported by the DataNode
and that have no blocks associated. These stale storages can skew the balancer behavior.
> Detailed discussion on HDFS-7575.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message