Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D0B3F200C44 for ; Mon, 13 Mar 2017 05:15:10 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id CF361160B87; Mon, 13 Mar 2017 04:15:10 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 24F5C160B77 for ; Mon, 13 Mar 2017 05:15:09 +0100 (CET) Received: (qmail 57588 invoked by uid 500); 13 Mar 2017 04:15:09 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 57574 invoked by uid 99); 13 Mar 2017 04:15:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Mar 2017 04:15:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 28394182319 for ; Mon, 13 Mar 2017 04:15:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.651 X-Spam-Level: X-Spam-Status: No, score=0.651 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id GW4uBkCtSqeO for ; Mon, 13 Mar 2017 04:15:07 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 31BE65FE3A for ; Mon, 13 Mar 2017 04:15:07 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 07020E087E for ; Mon, 13 Mar 2017 04:15:05 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 8A704243AC for ; Mon, 13 Mar 2017 04:15:04 +0000 (UTC) Date: Mon, 13 Mar 2017 04:15:04 +0000 (UTC) From: "Yiqun Lin (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-11514) ChooseRandom can potentially be optimized MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 13 Mar 2017 04:15:11 -0000 [ https://issues.apache.org/jira/browse/HDFS-11514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15906838#comment-15906838 ] Yiqun Lin commented on HDFS-11514: ---------------------------------- Thanks for this improvement, [~vagarychen]! Would you add a unit test to verify this optimization? We can just do a add/remove node operation after setting up initial nodes. Then verify if the counts is updated after change. If yes, this optimization is making sense. > ChooseRandom can potentially be optimized > ----------------------------------------- > > Key: HDFS-11514 > URL: https://issues.apache.org/jira/browse/HDFS-11514 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode > Reporter: Chen Liang > Assignee: Chen Liang > Attachments: HDFS-11514.001.patch > > > Based on the offline discussion, one potential improvement to the {{chooseRandomWithStorageType}} added in HDFS-11482 is that, currently given a node, the method iterates all its children to sum up the number of candidate datanodes. Since datanode status change is much less frequent than block placement request. It is more efficient to get rid of this iteration check, by probably maintaining another disk type counter map. This JIRA tracks (but not limited) this optimization. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org