Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 17 Sep 2015 18:57:05 +0000 (UTC)
From: "Daniel Templeton (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12852998.1438965546000.3945.1442516225773@Atlassian.JIRA>
In-Reply-To: <JIRA.12852998.1438965546000@Atlassian.JIRA>
References: <JIRA.12852998.1438965546000@Atlassian.JIRA>
 <JIRA.12852998.1438965546379@arcas>
Subject: [jira] [Commented] (HDFS-8873) throttle directoryScanner
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-8873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14804279#comment-14804279 ] 

Daniel Templeton commented on HDFS-8873:
----------------------------------------

[sigh]  Those tests pass for me locally, so can't say why they failed.

The whitespace error is interesting.  I changed line n in the patch.  Jenkins complained about the whitespace on line n+1.  I fixed the whitespace on line n+1 in the next patch.  Jenkins is now complaining about the whitespace on line n+2.  There is no issue on line n+3, so I could correct n+2 and be done, but at that point I've made whitespace changes on two lines that I didn't otherwise touch.  What's the accepted why to do it?  Fix the whitespace or ignore the error?

> throttle directoryScanner
> -------------------------
>
>                 Key: HDFS-8873
>                 URL: https://issues.apache.org/jira/browse/HDFS-8873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.7.1
>            Reporter: Nathan Roberts
>            Assignee: Daniel Templeton
>         Attachments: HDFS-8873.001.patch, HDFS-8873.002.patch, HDFS-8873.003.patch
>
>
> The new 2-level directory layout can make directory scans expensive in terms of disk seeks (see HDFS-8791) for details. 
> It would be good if the directoryScanner() had a configurable duty cycle that would reduce its impact on disk performance (much like the approach in HDFS-8617). 
> Without such a throttle, disks can go 100% busy for many minutes at a time (assuming the common case of all inodes in cache but no directory blocks cached, 64K seeks are required for full directory listing which translates to 655 seconds) 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)