hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11461) DataNode Disk Outlier Detection
Date Tue, 28 Feb 2017 20:41:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888830#comment-15888830

Arpit Agarwal commented on HDFS-11461:

Thanks [~hanishakoneru]. The patch looks good. A few more comments:
# diskOutliersStats should be updated atomically. Else if someone reads the map after the
{{clear}} call but before the for loop completes they will see partial results. You can allocate
a new map in {{detectAndUpdateDiskOutliers}} and replace the reference which will be atomic.
# We can move startDiskOutlierDetectionThread, detectAndUpdateDiskOutliers and diskOutliersStats
to a new package-private class. Also the shutdown logic can be moved to a method of the new
class. e.g. shutdownAndWait. Sorry I should have suggested this before.
# diskOutliersStats can be defined as a Map<String, Map<OpType, Double>>, so that
its explicit which values are read/write/meta.

Also once the logic is isolated to a new class, see if you can write a unit test (mock-based
may be easiest). Ok if you want to file a separate Jira to do that.

> DataNode Disk Outlier Detection
> -------------------------------
>                 Key: HDFS-11461
>                 URL: https://issues.apache.org/jira/browse/HDFS-11461
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs
>            Reporter: Hanisha Koneru
>            Assignee: Hanisha Koneru
>         Attachments: HDFS-11461.000.patch, HDFS-11461.001.patch
> Similar to how DataNodes collect peer performance statistics, we can collect disk performance
statistics per datanode and detect outliers among them, if any.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message