hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7433) DatanodeManager#datanodeMap should be a HashMap, not a TreeMap, to optimize lookup performance
Date Thu, 04 Dec 2014 04:09:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233895#comment-14233895
] 

Ming Ma commented on HDFS-7433:
-------------------------------

This is a good improvement.

Perhaps it is better call {{d.decommissioningStatus.setLastScan(scanNumber);}} for all nodes,
not just decomm-in-progress nodes. Otherwise, the NN could have nodes with much smaller number
than the current scan number after some time. If you decomm those nodes later, decomm check
might keep checking the same set of nodes until those nodes' scan numbers are the same as
the current scan number.

> DatanodeManager#datanodeMap should be a HashMap, not a TreeMap, to optimize lookup performance
> ----------------------------------------------------------------------------------------------
>
>                 Key: HDFS-7433
>                 URL: https://issues.apache.org/jira/browse/HDFS-7433
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.0.0-alpha, 3.0.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>            Priority: Critical
>         Attachments: HDFS-7433.patch, HDFS-7433.patch
>
>
> The datanode map is currently a {{TreeMap}}.  For many thousands of datanodes, tree lookups
are ~10X more expensive than a {{HashMap}}.  Insertions and removals are up to 100X more expensive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message