hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "BELUGA BEHR (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-9560) Fair AvailableSpaceVolumeChoosingPolicy
Date Wed, 16 Dec 2015 16:28:46 GMT

     [ https://issues.apache.org/jira/browse/HDFS-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

BELUGA BEHR updated HDFS-9560:
    Attachment: AvailableSpaceVolumeChoosingPolicy.java

> Fair AvailableSpaceVolumeChoosingPolicy
> ---------------------------------------
>                 Key: HDFS-9560
>                 URL: https://issues.apache.org/jira/browse/HDFS-9560
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: BELUGA BEHR
>            Priority: Minor
>         Attachments: AvailableSpaceVolumeChoosingPolicy.java
> I took a look at AvailableSpaceVolumeChoosingPolicy.  It seems a bit overkill and includes
some configuration items that seem a bit arbitrary with no real clear guidance on how to effectively
use them:
> _dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction_
> _dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold_
> I have created an alternative implementation that does not require any external configuration,
is thread-safe, and requires no synchronization.
> "Weighted Randomized Ordering"
> http://stackoverflow.com/questions/23971365/weighted-randomized-ordering
> Conceptually, a dart-board is constructed of several wedges, each wedge represents a
disk volume.  The more available space that a volume has relative to the other volumes, the
larger its wedge.  Then, a dart is thrown at the board and whichever wedge(volume) the dart
lands on, that wedge is assigned the incoming block.
> Over time, the wedges balance and all have an equal chance of being "hit."

This message was sent by Atlassian JIRA

View raw message