hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Roberts (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-8873) throttle directoryScanner
Date Fri, 07 Aug 2015 16:39:45 GMT
Nathan Roberts created HDFS-8873:

             Summary: throttle directoryScanner
                 Key: HDFS-8873
                 URL: https://issues.apache.org/jira/browse/HDFS-8873
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode
    Affects Versions: 2.7.1
            Reporter: Nathan Roberts

The new 2-level directory layout can make directory scans expensive in terms of disk seeks
(see HDFS-8791) for details. 

It would be good if the directoryScanner() had a configurable duty cycle that would reduce
its impact on disk performance (much like the approach in HDFS-8617). 

Without such a throttle, disks can go 100% busy for many minutes at a time (assuming the common
case of all inodes in cache but no directory blocks cached, 64K seeks are required for full
directory listing which translates to 655 seconds) 

This message was sent by Atlassian JIRA

View raw message