hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Surendra Singh Lilhore (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9034) "StorageTypeStats" Metric should not count failed storage.
Date Mon, 21 Dec 2015 09:46:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15066254#comment-15066254

Surendra Singh Lilhore commented on HDFS-9034:

bq. stats.subtract(node) will be called, but it will not remove the FAILED storage.
 Sorry if I understand wrong but I feel it will not happened. If we see the code
  synchronized void updateHeartbeat(final DatanodeDescriptor node,
      StorageReport[] reports, long cacheCapacity, long cacheUsed,
      int xceiverCount, int failedVolumes,
      VolumeFailureSummary volumeFailureSummary) {
    node.updateHeartbeat(reports, cacheCapacity, cacheUsed,
      xceiverCount, failedVolumes, volumeFailureSummary);
It will subtract before updating the FAILED state of ARCHIVE storage in {{node}} object. {{node}}
will contain the old state(NORMAL), new FAILED state is there in {{reports}}. After subtract
{{node.updateHeartbeat(…………)}} will update the new state in node object.

Same scenario I have done in test case, if you check test code from patch

+    DataNodeTestUtils.injectDataDirFailure(dn1ArcVol1);
+    DataNodeTestUtils.injectDataDirFailure(dn2ArcVol1);
+    DataNodeTestUtils.injectDataDirFailure(dn3ArcVol1);
+    // wait for heartbeat
+    Thread.sleep(6000);
+    storageTypeStatsMap = cluster.getNamesystem().getBlockManager()
+        .getStorageTypeStats();
+    assertFalse("StorageTypeStatsMap should not contain DISK Storage type",
+        storageTypeStatsMap.containsKey(StorageType.DISK));

after injecting DISK failure on all the DN's , DISK storage will not be there in {{storageTypeStatsMap}},
mean FAILED storage is subtracted.

> "StorageTypeStats" Metric should not count failed storage.
> ----------------------------------------------------------
>                 Key: HDFS-9034
>                 URL: https://issues.apache.org/jira/browse/HDFS-9034
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>            Reporter: Archana T
>            Assignee: Surendra Singh Lilhore
>         Attachments: HDFS-9034.01.patch, HDFS-9034.02.patch, HDFS-9034.03.patch, HDFS-9034.04.patch,
> When we remove one storage type from all the DNs, still NN UI shows entry of those storage
type --
> Ex:for ARCHIVE
> Steps--
> 1. ARCHIVE Storage type was added for all DNs
> 2. Stop DNs
> 3. Removed ARCHIVE Storages from all DNs
> 4. Restarted DNs
> NN UI shows below --
> DFS Storage Types
> Storage Type Configured Capacity Capacity Used Capacity Remaining 
> ARCHIVE	57.18 GB	64 KB (0%)	39.82 GB (69.64%)	64 KB	1

This message was sent by Atlassian JIRA

View raw message