hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1312) Re-balance disks within a Datanode
Date Fri, 18 Feb 2011 16:19:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996487#comment-12996487
] 

Steve Loughran commented on HDFS-1312:
--------------------------------------

HDFS-1362 and this issue are part of the HDFS-664 problem "support efficient hotswap". 
Before worrying about this one, consider HDFS-1121, which is provide a way to monitor the
distribution (i.e. web view). That web/management view would be how we'd test the rebalancing
works, so its a pre-req. Also it's best to keep the issues independent (where possible), so
worry about getting HDFS-1362 in first before trying to extend it. 

That said, because the #of HDDs/server is growing to 12 or more 2TB/unit, with 3TB on the
horizon, we will need this feature in the 0.23-0.24 timeframe.

> Re-balance disks within a Datanode
> ----------------------------------
>
>                 Key: HDFS-1312
>                 URL: https://issues.apache.org/jira/browse/HDFS-1312
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node
>            Reporter: Travis Crawford
>
> Filing this issue in response to ``full disk woes`` on hdfs-user.
> Datanodes fill their storage directories unevenly, leading to situations where certain
disks are full while others are significantly less used. Users at many different sites have
experienced this issue, and HDFS administrators are taking steps like:
> - Manually rebalancing blocks in storage directories
> - Decomissioning nodes & later readding them
> There's a tradeoff between making use of all available spindles, and filling disks at
the sameish rate. Possible solutions include:
> - Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy
environments this will still make use of all spindles, equalizing disk use over time.
> - Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced
in older cluster nodes.
> Datanodes should actively manage their local disk so operator intervention is not needed.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message