hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8989) Allow rate-limiting the DirectoryScanner via time-slicing
Date Thu, 03 Sep 2015 02:10:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728394#comment-14728394

Colin Patrick McCabe commented on HDFS-8989:

Yeah, I didn't see HDFS-8873 when I filed this.  Thanks.

> Allow rate-limiting the DirectoryScanner via time-slicing
> ---------------------------------------------------------
>                 Key: HDFS-8989
>                 URL: https://issues.apache.org/jira/browse/HDFS-8989
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.0.6-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Daniel Templeton
> When the inodes and directory entries for HDFS block and meta files are not in the cache,
the DirectoryScanner can create enormous disk load via generating many seeks.  This can effectively
make the disks unusable for other purposes.  We should allow rate-limiting the DirectoryScanner
via time-slicing, so that it only runs every 100 ms out of 1 s (or whatever other percentage
we configure.)

This message was sent by Atlassian JIRA

View raw message