Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3A52F187D8 for ; Tue, 18 Aug 2015 01:33:47 +0000 (UTC) Received: (qmail 64570 invoked by uid 500); 18 Aug 2015 01:33:47 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 64496 invoked by uid 500); 18 Aug 2015 01:33:46 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 64482 invoked by uid 99); 18 Aug 2015 01:33:46 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 18 Aug 2015 01:33:46 +0000 Date: Tue, 18 Aug 2015 01:33:46 +0000 (UTC) From: "Yi Liu (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-8862) BlockManager#excessReplicateMap should use a HashMap MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14700577#comment-14700577 ] Yi Liu commented on HDFS-8862: ------------------------------ One more discussion, do you think it's worth to extend java HashMap and implement the {{shrink}}? Since it's better to have the shrinked HashMap in some places. From my point of review, I think it's worth and for quick glance, seems few code is needed. > BlockManager#excessReplicateMap should use a HashMap > ---------------------------------------------------- > > Key: HDFS-8862 > URL: https://issues.apache.org/jira/browse/HDFS-8862 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode > Reporter: Yi Liu > Assignee: Yi Liu > Fix For: 2.8.0 > > Attachments: HDFS-8862.001.patch > > > Per [~cmccabe]'s comments in HDFS-8792, this JIRA is to discuss improving {{BlockManager#excessReplicateMap}}. > That's right HashMap don't ever shrink when elements are removed, but TreeMap entry needs to store more (memory) references (left, right, parent) than HashMap entry (only one reference next), even when there is element removing and cause some entry empty, the empty HashMap entry is just a {{null}} reference (4 bytes), so they are close at this point. On the other hand, the key of {{excessReplicateMap}} is datanode uuid, so the entries number is almost fixed, so HashMap memory is good than TreeMap memory in this case. I think the most important is the search/insert/remove performance, HashMap is absolutely better than TreeMap. Because we don't need to sort, we should use HashMap instead of TreeMap -- This message was sent by Atlassian JIRA (v6.3.4#6332)