hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11461) DataNode Disk Outlier Detection
Date Tue, 28 Feb 2017 00:39:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886922#comment-15886922

Arpit Agarwal commented on HDFS-11461:

[~hanishakoneru], a few comments:
# {{DataNode#shutdown}} should use slowDiskDetectionThread.join() instead of sleep.
# {{diskOutliers}} should maintain the mean read/write/meta latency for each flagged disk.
# The low threshold should be higher than 1ms (seek latency of a 7200 RPM disk is 4ms). Let's
conservatively set this to 20ms.
# startDiskOutlierDetectionThread should call Thread.currentThread().interrupt() after catching
InterruptedException. See https://www.ibm.com/developerworks/library/j-jtp05236/
# slowDiskDetectionThread should be a daemon thread.
# DataNodePeerMetrics should also use OutlierDetector constructor that accepts minNumResources,
and pass {{10}}, to keep the behavior consistent with what we have.

> DataNode Disk Outlier Detection
> -------------------------------
>                 Key: HDFS-11461
>                 URL: https://issues.apache.org/jira/browse/HDFS-11461
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs
>            Reporter: Hanisha Koneru
>            Assignee: Hanisha Koneru
>         Attachments: HDFS-11461.000.patch
> Similar to how DataNodes collect peer performance statistics, we can collect disk performance
statistics per datanode and detect outliers among them, if any.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message