hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1804) Modify or add a new block-volume device choosing policy that looks at free space
Date Fri, 05 Apr 2013 05:36:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13623374#comment-13623374

Chris Nauroth commented on HDFS-1804:

Hi, Aaron.  This looks really good.  I applied the patch locally and ran the new tests successfully.
 Here are a couple of thoughts:

# The algorithm doesn't apply a scaling factor to account for a heterogeneous set of volumes.
 For example, consider a mix of 500 GB disks and 2 TB disks, all initially empty.  It would
favor the 2 TB disks heavily until the amount of free space across all volumes (both 500 GB
and 2 TB) balanced out.  This would mean that the risk of harming concurrency is higher if
using heterogeneous volumes.  Another scaling factor for relative volume capacity could account
for this.  I defer to you on whether or not to consider this in scope right now or skip it
and treat it as a potential future feature request to be handled in a different jira.  The
scope that I see in the jira description is open-ended.  I don't think you can even get total
capacity through {{FsVolumeSpi}} right now, so I expect it would be a much bigger change.
# Minor thing: the new helper methods in {{GenericTestUtils}} would have greater symmetry
with JUnit's built-in assert methods if the order of arguments was: expected, actual, delta.
# For initialization of {{balancedPreferencePercent}}:

  private float balancedPreferencePercent = 0.75f;

here?  (Though I imagine it won't really matter in practice since it gets overwritten from

Thanks, Aaron!

> Modify or add a new block-volume device choosing policy that looks at free space
> --------------------------------------------------------------------------------
>                 Key: HDFS-1804
>                 URL: https://issues.apache.org/jira/browse/HDFS-1804
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Harsh J
>            Assignee: Aaron T. Myers
>            Priority: Minor
>         Attachments: HDFS-1804.patch, HDFS-1804.patch, HDFS-1804.patch, HDFS-1804.patch
> HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the
vanilla RoundRobin as its default.
> An additional implementation that also takes into consideration the free space remaining
on the disk (or other params) should be a good addition as an alternative to vanilla RR.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message