hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1804) Modify or add a new block-volume device choosing policy that looks at free space
Date Fri, 05 Apr 2013 05:36:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13623374#comment-13623374
] 

Chris Nauroth commented on HDFS-1804:
-------------------------------------

Hi, Aaron.  This looks really good.  I applied the patch locally and ran the new tests successfully.
 Here are a couple of thoughts:

# The algorithm doesn't apply a scaling factor to account for a heterogeneous set of volumes.
 For example, consider a mix of 500 GB disks and 2 TB disks, all initially empty.  It would
favor the 2 TB disks heavily until the amount of free space across all volumes (both 500 GB
and 2 TB) balanced out.  This would mean that the risk of harming concurrency is higher if
using heterogeneous volumes.  Another scaling factor for relative volume capacity could account
for this.  I defer to you on whether or not to consider this in scope right now or skip it
and treat it as a potential future feature request to be handled in a different jira.  The
scope that I see in the jira description is open-ended.  I don't think you can even get total
capacity through {{FsVolumeSpi}} right now, so I expect it would be a much bigger change.
# Minor thing: the new helper methods in {{GenericTestUtils}} would have greater symmetry
with JUnit's built-in assert methods if the order of arguments was: expected, actual, delta.
# For initialization of {{balancedPreferencePercent}}:

{code}
  private float balancedPreferencePercent = 0.75f;
{code}

Do you want to use {{DFSConfigKeys#DFS_DATANODE_FSDATASET_VOLUME_CHOOSING_BALANCED_SPACE_PREFERENCE_PERCENT_DEFAULT}}
here?  (Though I imagine it won't really matter in practice since it gets overwritten from
config.)


Thanks, Aaron!

                
> Modify or add a new block-volume device choosing policy that looks at free space
> --------------------------------------------------------------------------------
>
>                 Key: HDFS-1804
>                 URL: https://issues.apache.org/jira/browse/HDFS-1804
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Harsh J
>            Assignee: Aaron T. Myers
>            Priority: Minor
>         Attachments: HDFS-1804.patch, HDFS-1804.patch, HDFS-1804.patch, HDFS-1804.patch
>
>
> HDFS-1120 introduced pluggable block-volume choosing policies, but still carries the
vanilla RoundRobin as its default.
> An additional implementation that also takes into consideration the free space remaining
on the disk (or other params) should be a good addition as an alternative to vanilla RR.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message