Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 10 Oct 2016 23:07:20 +0000 (UTC)
From: "Zhe Zhang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13010041.1475712419000.788282.1476140840680@Atlassian.JIRA>
In-Reply-To: <JIRA.13010041.1475712419000@Atlassian.JIRA>
References: <JIRA.13010041.1475712419000@Atlassian.JIRA> <JIRA.13010041.1475712419276@arcas>
Subject: [jira] [Commented] (HDFS-10967) Add configuration for
 BlockPlacementPolicy to avoid near-full DataNodes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 10 Oct 2016 23:07:22 -0000


    [ https://issues.apache.org/jira/browse/HDFS-10967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563835#comment-15563835 ] 

Zhe Zhang commented on HDFS-10967:
----------------------------------

I verified reported test failures and could only reproduce {{TestHdfsConfigFields}}. Once we agree on the overall structure I'll do the due diligence:
# Add the item to {{hdfs-default.xml}}
# Document the new dfsAdmin command
# Clear up the checkStyle warnings

> Add configuration for BlockPlacementPolicy to avoid near-full DataNodes
> -----------------------------------------------------------------------
>
>                 Key: HDFS-10967
>                 URL: https://issues.apache.org/jira/browse/HDFS-10967
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>              Labels: balancer
>         Attachments: HDFS-10967.00.patch, HDFS-10967.01.patch, HDFS-10967.02.patch, HDFS-10967.03.patch
>
>
> Large production clusters are likely to have heterogeneous nodes in terms of storage capacity, memory, and CPU cores. It is not always possible to proportionally ingest data into DataNodes based on their remaining storage capacity. Therefore it's possible for a subset of DataNodes to be much closer to full capacity than the rest.
> This heterogeneity is most likely rack-by-rack -- i.e. _m_ whole racks of low-storage nodes and _n_ whole racks of high-storage nodes. So It'd be very useful if we can lower the chance for those near-full DataNodes to become destinations for the 2nd and 3rd replicas.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org