hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7518) Heartbeat processing doesn't have to take FSN readLock
Date Sat, 13 Dec 2014 07:18:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14245232#comment-14245232
] 

Haohui Mai commented on HDFS-7518:
----------------------------------

Currently the FSN lock implements mutual exclusion for both {{FSNamesystem}} and {{BlockManager}}.
More importantly, it also implements mutual exclusions when updating individual {{BlockInfo}}
objects.

I think that as a first step, we can move the workflow of managing datanode out of the FSN
lock, but keeping things like populating replication queues in the lock.

In the slightly longer term I think we need to work on HDFS-7437 to decouple the implicit
dependency introduced by {{BlockInfo}} objects, and gradually to decouple other functionalities
and the FSN lock.

> Heartbeat processing doesn't have to take FSN readLock
> ------------------------------------------------------
>
>                 Key: HDFS-7518
>                 URL: https://issues.apache.org/jira/browse/HDFS-7518
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Ming Ma
>
> NameNode takes global read lock when it process heartbeat RPCs from DataNodes. This increases
lock contention and could impact NN overall throughput. Given Heartbeat processing needs to
access data specific to the DataNode that invokes the RPC; it could just synchronize on the
specific DataNode and datanodeMap.
> It looks like each DatanodeDescriptor already keeps its own recover blocks, replication
blocks and invalidate blocks. There are several places that needed to be changed to remove
FSN lock.
> As mentioned in other jiras, we need to some mechanism to reason about the correctness
of the solution.
> Thoughts?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message