hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "He Liangliang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9422) Unnecessary FsDatasetImpl locking in DirectoryScanner cause periodic datanode pauses
Date Mon, 23 Nov 2015 12:07:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022007#comment-15022007
] 

He Liangliang commented on HDFS-9422:
-------------------------------------

Just record the time consumed, jstack not captured. ~5s(mean around 2s and max 5s) is indeed
longer than expected. Just made a local test, sorting 500k blocks take 500ms and 5m takes
5s. So it's still a reasonable time considering the contention with cpu-loaded regionserver
processes in the same box.

> Unnecessary FsDatasetImpl locking in DirectoryScanner cause periodic datanode pauses
> ------------------------------------------------------------------------------------
>
>                 Key: HDFS-9422
>                 URL: https://issues.apache.org/jira/browse/HDFS-9422
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: He Liangliang
>            Assignee: He Liangliang
>         Attachments: HDFS-9422.patch
>
>
> In DirectoryScanner, the scan call hold the global dataset lock for quite long time (typically
> 5 secs for 500k blocks). This will stuck the client which need acquire this lock. For
applications like HBase, this will affect the latency. In fact, this lock is unnecessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message