hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Templeton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8873) throttle directoryScanner
Date Wed, 23 Sep 2015 16:11:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904730#comment-14904730
] 

Daniel Templeton commented on HDFS-8873:
----------------------------------------

Forgot to mention that this patch tries be fair about oversleeps when throttling, but it doesn't
do anything to credit for full seconds lost entirely to oversleeps.  For example, if the throttle
is set to 100ms, the following could happen:

.0000 - thread calls throttle(), no block
.0422 - thread calls throttle(), sleep for 588ms
.5016 - thread wakes up from oversleep, run limit this second set to 116ms
.5219 - thread calls throttle(), sleep for 781ms

In the second that the thread wakes up from the oversleep, we try to give it its full run
time, but we don't credit it for full seconds lost.  Notice also that threads only offer to
block once per cycle, so setting a very low throttle limit will just make the threads run
one cycle between sleeps.

The following could also happen:

.0000 - thread calls throttle(), no block
.0422 - thread calls throttle(), sleep for 588ms
.1970 - thread wakes up from oversleep, run limit this second set to 1070ms
.2089 - thread calls throttle(), no block
.2101 - thread calls throttle(), sleep for 899ms

When the thread wakes up from the oversleep, we give it more than a second's worth of run
time because it's so close to the end of the second.  (Any value over 999 just means not to
throttle the thread that second.)  The next second starts the count over, and at .2089, even
though it's been running for over 100 consecutive ms, we don't block it, because it hasn't
run for 100ms in this second yet.  When it checks in the next time, at .2101, we see it's
over the 100ms limit and block it for the rest of the second.

> throttle directoryScanner
> -------------------------
>
>                 Key: HDFS-8873
>                 URL: https://issues.apache.org/jira/browse/HDFS-8873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.7.1
>            Reporter: Nathan Roberts
>            Assignee: Daniel Templeton
>         Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch, HDFS-8873.003.patch, HDFS-8873.004.patch,
HDFS-8873.005.patch, HDFS-8873.006.patch
>
>
> The new 2-level directory layout can make directory scans expensive in terms of disk
seeks (see HDFS-8791) for details. 
> It would be good if the directoryScanner() had a configurable duty cycle that would reduce
its impact on disk performance (much like the approach in HDFS-8617). 
> Without such a throttle, disks can go 100% busy for many minutes at a time (assuming
the common case of all inodes in cache but no directory blocks cached, 64K seeks are required
for full directory listing which translates to 655 seconds) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message