hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8989) Allow rate-limiting the DirectoryScanner via time-slicing
Date Tue, 01 Sep 2015 23:45:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726434#comment-14726434

Andrew Wang commented on HDFS-8989:

This looks like a dupe of HDFS-8873 which has more watchers; want to take it there?

> Allow rate-limiting the DirectoryScanner via time-slicing
> ---------------------------------------------------------
>                 Key: HDFS-8989
>                 URL: https://issues.apache.org/jira/browse/HDFS-8989
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.0.6-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Daniel Templeton
> When the inodes and directory entries for HDFS block and meta files are not in the cache,
the DirectoryScanner can create enormous disk load via generating many seeks.  This can effectively
make the disks unusable for other purposes.  We should allow rate-limiting the DirectoryScanner
via time-slicing, so that it only runs every 100 ms out of 1 s (or whatever other percentage
we configure.)

This message was sent by Atlassian JIRA

View raw message