hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathan Roberts (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-8873) throttle directoryScanner
Date Fri, 07 Aug 2015 16:39:45 GMT
Nathan Roberts created HDFS-8873:
------------------------------------

             Summary: throttle directoryScanner
                 Key: HDFS-8873
                 URL: https://issues.apache.org/jira/browse/HDFS-8873
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode
    Affects Versions: 2.7.1
            Reporter: Nathan Roberts


The new 2-level directory layout can make directory scans expensive in terms of disk seeks
(see HDFS-8791) for details. 

It would be good if the directoryScanner() had a configurable duty cycle that would reduce
its impact on disk performance (much like the approach in HDFS-8617). 

Without such a throttle, disks can go 100% busy for many minutes at a time (assuming the common
case of all inodes in cache but no directory blocks cached, 64K seeks are required for full
directory listing which translates to 655 seconds) 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message