Return-Path: X-Original-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B818117D2B for ; Wed, 1 Apr 2015 23:32:01 +0000 (UTC) Received: (qmail 42149 invoked by uid 500); 1 Apr 2015 23:31:54 -0000 Delivered-To: apmail-hadoop-hdfs-dev-archive@hadoop.apache.org Received: (qmail 42012 invoked by uid 500); 1 Apr 2015 23:31:54 -0000 Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-dev@hadoop.apache.org Received: (qmail 41773 invoked by uid 99); 1 Apr 2015 23:31:54 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2015 23:31:54 +0000 Date: Wed, 1 Apr 2015 23:31:54 +0000 (UTC) From: "Kihwal Lee (JIRA)" To: hdfs-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Kihwal Lee created HDFS-8041: -------------------------------- Summary: Consider remaining space during block blockplacement if dfs space is highly utilized Key: HDFS-8041 URL: https://issues.apache.org/jira/browse/HDFS-8041 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kihwal Lee This feature is helpful in avoiding smaller nodes (i.e. heterogeneous environment) getting constantly being full when the overall space utilization is over a certain threshold. When the utilization is low, balancer can keep up, but once the average per-node byte goes over the capacity of the smaller nodes, they get full so quickly even after perfect balance. This jira proposes an improvement that can be optionally enabled in order to slow down the rate of space usage growth of smaller nodes if the overall storage utilization is over a configured threshold. It will not replace balancer, rather will help balancer keep up. Also, the primary replica placement will not be affected. Only the replicas typically placed in a remote rack will be subject to this check. The appropriate threshold is cluster configuration specific. There is no generally good value to set, thus it is disabled by default. We have seen cases where the threshold of 85% - 90% would help. Figuring when {{totalSpaceUsed / numNodes}} becomes close to the capacity of a smaller node is helpful in determining the threshold. -- This message was sent by Atlassian JIRA (v6.3.4#6332)