hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1312) Re-balance disks within a Datanode
Date Tue, 22 Dec 2015 23:32:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068911#comment-15068911
] 

Lei (Eddy) Xu commented on HDFS-1312:
-------------------------------------

Hey, [~anu].

Thanks a lot for working on this.  It is a very demanding feature.

I have a few questions regarding the design, much appreciated if you can answer them to help
me to better understand the design.

* Will disk balancer be a daemon or a CLI tool, that waits the process to finishes?  
* Where is the Planner executed? A DiskBalancer daemon / CLI tool or NN? 
* When copying a replica from one volume to another, how to prevent it to have concurrent
issues with the {{DirectoryScanner}} in background?
* Is there a limitation of how many such disk balancer jobs can run in the cluster? Could
other job queries the status of a running job? 

Thanks again!
 

> Re-balance disks within a Datanode
> ----------------------------------
>
>                 Key: HDFS-1312
>                 URL: https://issues.apache.org/jira/browse/HDFS-1312
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode
>            Reporter: Travis Crawford
>            Assignee: Anu Engineer
>         Attachments: Architecture_and_testplan.pdf, disk-balancer-proposal.pdf
>
>
> Filing this issue in response to ``full disk woes`` on hdfs-user.
> Datanodes fill their storage directories unevenly, leading to situations where certain
disks are full while others are significantly less used. Users at many different sites have
experienced this issue, and HDFS administrators are taking steps like:
> - Manually rebalancing blocks in storage directories
> - Decomissioning nodes & later readding them
> There's a tradeoff between making use of all available spindles, and filling disks at
the sameish rate. Possible solutions include:
> - Weighting less-used disks heavier when placing new blocks on the datanode. In write-heavy
environments this will still make use of all spindles, equalizing disk use over time.
> - Rebalancing blocks locally. This would help equalize disk use as disks are added/replaced
in older cluster nodes.
> Datanodes should actively manage their local disk so operator intervention is not needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message