hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1312) Re-balance disks within a Datanode
Date Wed, 26 Sep 2012 09:14:07 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463642#comment-13463642
] 

Steve Loughran commented on HDFS-1312:
--------------------------------------

I don't think it's a wontfix, just that nobody has sat down to fix it. With the trend towards
very storage intense servers (12-16x 3TBs), this problem has grown -and it does need fixing


What does it take?
# tests
# solution
# patch submit and review.

It may be possible to implement the rebalance operation as some script (Please, not bash,
something better like Python); ops could run this script after installing a new HDD. 

A way to assess HDD imbalance in a DN or across the cluster would be good too; that could
pick up problems which manual/automated intervention could fix.

BTW,, not everyone splits MR and DFS onto separate disk partitions, as that remove flexibility
and the ability to handle large-intermediate-output MR jobs. 

                
> Re-balance disks within a Datanode
> ----------------------------------
>
>                 Key: HDFS-1312
>                 URL: https://issues.apache.org/jira/browse/HDFS-1312
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node
>            Reporter: Travis Crawford
>
> Filing this issue in response to ``full disk woes`` on hdfs-user.
> Datanodes fill their storage directories unevenly, leading to situations where certain
disks are full while others are significantly less used. Users at many different sites have
experienced this issue, and HDFS administrators are taking steps like:
> - Manually rebalancing blocks in storage directories
> - Decomissioning nodes & later readding them
> There's a tradeoff between making use of all available spindles, and filling disks at
the sameish rate. Possible solutions include:
> - Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy
environments this will still make use of all spindles, equalizing disk use over time.
> - Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced
in older cluster nodes.
> Datanodes should actively manage their local disk so operator intervention is not needed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message