hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9422) Unnecessary FsDatasetImpl locking in DirectoryScanner cause periodic datanode pauses
Date Tue, 17 Nov 2015 17:11:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15009046#comment-15009046
] 

stack commented on HDFS-9422:
-----------------------------

Patch just undoes the synchronize on dataset.  This dataset synchronize in scan has been in
place for years. 

As is, there are 'holes' given we let go of the dataset lock after we finish scan and before
we go into the checkAndUpdate. The synchronize on dataset in scan seems gratuitous. The synchronize
in checkAndUpdate seems sufficient safety doing reconcile of memory and disk state.

Patch looks good to me.



> Unnecessary FsDatasetImpl locking in DirectoryScanner cause periodic datanode pauses
> ------------------------------------------------------------------------------------
>
>                 Key: HDFS-9422
>                 URL: https://issues.apache.org/jira/browse/HDFS-9422
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: He Liangliang
>            Assignee: He Liangliang
>         Attachments: HDFS-9422.patch
>
>
> In DirectoryScanner, the scan call hold the global dataset lock for quite long time (typically
> 5 secs for 500k blocks). This will stuck the client which need acquire this lock. For
applications like HBase, this will affect the latency. In fact, this lock is unnecessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message