hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8873) throttle directoryScanner
Date Wed, 23 Sep 2015 19:47:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905122#comment-14905122

Colin Patrick McCabe commented on HDFS-8873:

I guess the short summary is that I'm not happy with the "modulo" solution to the timekeeping
problem since I think its inaccuracy (in the direction of not running often enough, or at
all) will be unacceptable at lower throttle rates.

We have a StopWatch class which should provide a pretty easy solution to this problem that
doesn't suffer from these issues.  Just start a StopWatch when you enable scanning.  Check
the elapsed duration on the StopWatch before scanning each block-- if it is above dfs.datanode.directoryscan.throttle.limit.ms.per.sec,
then disable scanning and go into the other loop, which calls Thread.sleep until 1000 - dfs.datanode.directoryscan.throttle.limit.ms.per.sec
ms have elapsed.

This solution is robust against slightly shorter or longer thread pauses.  Like the module
solution, it also may run at a slightly lower rate than expected in the case where we oversleep.
 Unlike the modulo solution, it does not have a pathological case where we never get to run
at all despite a non-zero throttle rate.

399  public static final int     DFS_DATANODE_DIRECTORYSCAN_INTERVAL_DEFAULT = 21600;
460	    int defaultLimit =
462	    String logMsg;
464	    if (throttleLimitMsPerSec < defaultLimit) {
465	      logMsg = String.format(START_MESSAGE_WITH_THROTTLE, firstScanTime,
466	          scanPeriodMsecs, throttleLimitMsPerSec);
467	    } else {
468	      logMsg = String.format(START_MESSAGE, firstScanTime, scanPeriodMsecs);
469	    }
I'm confused about this code... why are we comparing the directory scan interval (in seconds)
with the throttle limit in milliseconds?  It seems like this was intended to test for the
case where the throttle is disabled, but some wires got crossed?

> throttle directoryScanner
> -------------------------
>                 Key: HDFS-8873
>                 URL: https://issues.apache.org/jira/browse/HDFS-8873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.7.1
>            Reporter: Nathan Roberts
>            Assignee: Daniel Templeton
>         Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch, HDFS-8873.003.patch, HDFS-8873.004.patch,
HDFS-8873.005.patch, HDFS-8873.006.patch
> The new 2-level directory layout can make directory scans expensive in terms of disk
seeks (see HDFS-8791) for details. 
> It would be good if the directoryScanner() had a configurable duty cycle that would reduce
its impact on disk performance (much like the approach in HDFS-8617). 
> Without such a throttle, disks can go 100% busy for many minutes at a time (assuming
the common case of all inodes in cache but no directory blocks cached, 64K seeks are required
for full directory listing which translates to 655 seconds) 

This message was sent by Atlassian JIRA

View raw message