hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "BELUGA BEHR (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-9560) Fair AvailableSpaceVolumeChoosingPolicy
Date Wed, 16 Dec 2015 16:26:46 GMT
BELUGA BEHR created HDFS-9560:
---------------------------------

             Summary: Fair AvailableSpaceVolumeChoosingPolicy
                 Key: HDFS-9560
                 URL: https://issues.apache.org/jira/browse/HDFS-9560
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: BELUGA BEHR
            Priority: Minor


I took a look at AvailableSpaceVolumeChoosingPolicy.  It seems a bit overkill and includes
some configuration items that seem a bit arbitrary with no real clear guidance on how to effectively
use them:

_dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction_
_dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold_

I have created an alternative implementation that does not require any external configuration,
is thread-safe, and requires no synchronization.

"Weighted Randomized Ordering"

http://stackoverflow.com/questions/23971365/weighted-randomized-ordering

Conceptually, a dart-board is constructed of several wedges, each wedge represents a disk
volume.  The more available space that a volume has relative to the other volumes, the larger
its wedge.  Then, a dart is thrown at the board and whichever wedge(volume) the dart lands
on, that wedge is assigned the incoming block.

Over time, the wedges balance and all have an equal chance of being "hit."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message