Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 02C10106E0 for ; Thu, 14 May 2015 13:06:02 +0000 (UTC) Received: (qmail 87824 invoked by uid 500); 14 May 2015 13:06:01 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 87765 invoked by uid 500); 14 May 2015 13:06:01 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 87753 invoked by uid 99); 14 May 2015 13:06:01 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 May 2015 13:06:01 +0000 Date: Thu, 14 May 2015 13:06:01 +0000 (UTC) From: "Liu Shaohui (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-8131) Implement a space balanced block placement policy MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-8131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543640#comment-14543640 ] Liu Shaohui commented on HDFS-8131: ----------------------------------- The failed test: TestHdfsConfigFields has no relations with this patch. It failed in trunk also. > Implement a space balanced block placement policy > ------------------------------------------------- > > Key: HDFS-8131 > URL: https://issues.apache.org/jira/browse/HDFS-8131 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode > Affects Versions: 3.0.0 > Reporter: Liu Shaohui > Assignee: Liu Shaohui > Priority: Minor > Labels: BlockPlacementPolicy > Fix For: 3.0.0 > > Attachments: HDFS-8131-v1.diff, HDFS-8131-v2.diff, HDFS-8131-v3.diff, balanced.png > > > The default block placement policy will choose datanodes for new blocks randomly, which will result in unbalanced space used percent among datanodes after an cluster expansion. The old datanodes always are in high used percent of space and new added ones are in low percent. > Through we can used the external balance tool to balance the space used rate, it will cost extra network IO and it's not easy to control the balance speed. > An easy solution is to implement an balanced block placement policy which will choose low used percent datanodes for new blocks with a little high possibility. In a not long term, the used percent of datanodes will trend to be balanced. > Suggestions and discussions are welcomed. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)