Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 4 Mar 2011 08:53:37 +0000 (UTC)
From: "Allen Wittenauer (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: 
 <2124533360.13330.1299228817022.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Commented: (HDFS-1312) Re-balance disks within a Datanode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002509#comment-13002509 ] 

Allen Wittenauer commented on HDFS-1312:
----------------------------------------

>For the cluster monitoring issue, we could not expect an integrated monitoring 
> substitute the external monitoring system. Thus, we need to make clear
> what information gathering requirements should be considered inside hdfs.

There is nothing preventing us from building a jsp that runs on the datanode that shows the file systems in use and the relevant stats for those fs's.  

>And the "lock" I mentions is to stop write new blocks into the volume,
> which is the simplest way to migrate blocks. 

This is going to be a big performance hit.  I think the focus should be on already written/closed blocks and just ignore new blocks.


> Re-balance disks within a Datanode
> ----------------------------------
>
>                 Key: HDFS-1312
>                 URL: https://issues.apache.org/jira/browse/HDFS-1312
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node
>            Reporter: Travis Crawford
>
> Filing this issue in response to ``full disk woes`` on hdfs-user.
> Datanodes fill their storage directories unevenly, leading to situations where certain disks are full while others are significantly less used. Users at many different sites have experienced this issue, and HDFS administrators are taking steps like:
> - Manually rebalancing blocks in storage directories
> - Decomissioning nodes & later readding them
> There's a tradeoff between making use of all available spindles, and filling disks at the sameish rate. Possible solutions include:
> - Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy environments this will still make use of all spindles, equalizing disk use over time.
> - Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced in older cluster nodes.
> Datanodes should actively manage their local disk so operator intervention is not needed.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira