hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9422) Unnecessary FsDatasetImpl locking in DirectoryScanner cause periodic datanode pauses
Date Fri, 13 Nov 2015 13:37:10 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003986#comment-15003986
] 

Kai Zheng commented on HDFS-9422:
---------------------------------

Hi [~heliangliang],

Thanks for finding and reporting the issue. I took a look at your patch and I agree we'll
need a better lock/sync scheme here, since the scan of disk may take much longer time than
previously assumed and holding of the global lock during the time will affect other operations.
However, some sync mechanism may be still needed because otherwise the diff computation result
may be out of sync or incorrect if the *dataset* block map is changed during the time. We
may limit the lock scope or use a fine-grained lock instead. 

> Unnecessary FsDatasetImpl locking in DirectoryScanner cause periodic datanode pauses
> ------------------------------------------------------------------------------------
>
>                 Key: HDFS-9422
>                 URL: https://issues.apache.org/jira/browse/HDFS-9422
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: He Liangliang
>            Assignee: He Liangliang
>         Attachments: HDFS-9422.patch
>
>
> In DirectoryScanner, the scan call hold the global dataset lock for quite long time (typically
> 5 secs for 500k blocks). This will stuck the client which need acquire this lock. For
applications like HBase, this will affect the latency. In fact, this lock is unnecessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message