hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7396) Revisit synchronization in Namenode
Date Sat, 06 Dec 2014 01:04:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14236420#comment-14236420
] 

Ming Ma commented on HDFS-7396:
-------------------------------

It might be nice if we can enforce the correctness via unit tests. For example, https://issues.apache.org/jira/browse/HDFS-7182
tries to assert any JMX metrics shouldn't require FSNamesystem read lock with the modified
unit test.

Federated NN is sort of orthogonal to this discussion given it is still useful to improve
the overall synchronization in NN even for federated NN. However it does make synchronization
improvement less critical; e.g., you can get the NN JVM heap size under 32G and use compressed
pointer, etc. There is still improvement we need to make to federated NN, such as dynamic
reallocation amount namespaces, have each NN host several volumes for efficient move, etc.
It will be interesting to understand if there is any plan to scale HDFS in that direction.

> Revisit synchronization in Namenode
> -----------------------------------
>
>                 Key: HDFS-7396
>                 URL: https://issues.apache.org/jira/browse/HDFS-7396
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>
> HDFS-2106 separated block management to a new package from namenode.  As part of it,
some code was refactored to new classes such as DatanodeManager, HeartbeatManager, etc.  There
are opportunities for improve locking in namenode while currently the synchronization in namenode
is mainly done by a single global FSNamesystem lock. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message