hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rushabh S Shah (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9087) Add some jitter to DataNode.checkDiskErrorThread
Date Wed, 16 Sep 2015 13:49:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14768959#comment-14768959
] 

Rushabh S Shah commented on HDFS-9087:
--------------------------------------

[~eclark]: Thanks for reporting the issue.
You might want to look at HDFS-8845.
Before that patch, the datanode was traversing all the subdirectories under each volume and
was trying to create a file.
But in that patch, it will only scan the finalized directory (not the subdirectory), rbw and
temp directory.
That should not make the disk busy even the scan is going across all the volumes at the same
time.

> Add some jitter to DataNode.checkDiskErrorThread
> ------------------------------------------------
>
>                 Key: HDFS-9087
>                 URL: https://issues.apache.org/jira/browse/HDFS-9087
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.6.0
>            Reporter: Elliott Clark
>            Assignee: Elliott Clark
>         Attachments: HDFS-9087-v0.patch, HDFS-9087-v1.patch
>
>
> If all datanodes are started across a cluster at the same time (or errors in the network
cause ioexceptions) there can be storms where lots of datanodes check their disks at the exact
same time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message