Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 11 Sep 2015 02:34:46 +0000 (UTC)
From: "David Kaiser (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12469869.1279755967000.309412.1441938886140@Atlassian.JIRA>
In-Reply-To: <JIRA.12469869.1279755967000@Atlassian.JIRA>
References: <JIRA.12469869.1279755967000@Atlassian.JIRA>
 <JIRA.12469869.1279755967333@arcas>
Subject: [jira] [Commented] (HDFS-1312) Re-balance disks within a Datanode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740038#comment-14740038 ] 

David Kaiser commented on HDFS-1312:
------------------------------------

Anu, I would like to add a comment to the tool proposal.  Could an administrator specify that a particular volume (or list of volumes) end up with exactly zero blocks, effectively emptying that volume from block storage, say to move blocks to support the task of "decommissioning a drive".

Intent of this would be that all blocks from the specified volume would be moved, as proposed by the DiskBalancer logic to the other volumes so that the then-empty volume could be removed from the dfs.datanode.dir path mapping.

> Re-balance disks within a Datanode
> ----------------------------------
>
>                 Key: HDFS-1312
>                 URL: https://issues.apache.org/jira/browse/HDFS-1312
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode
>            Reporter: Travis Crawford
>         Attachments: disk-balancer-proposal.pdf
>
>
> Filing this issue in response to ``full disk woes`` on hdfs-user.
> Datanodes fill their storage directories unevenly, leading to situations where certain disks are full while others are significantly less used. Users at many different sites have experienced this issue, and HDFS administrators are taking steps like:
> - Manually rebalancing blocks in storage directories
> - Decomissioning nodes & later readding them
> There's a tradeoff between making use of all available spindles, and filling disks at the sameish rate. Possible solutions include:
> - Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy environments this will still make use of all spindles, equalizing disk use over time.
> - Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced in older cluster nodes.
> Datanodes should actively manage their local disk so operator intervention is not needed.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)