hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11295) Check storage remaining instead of node remaining in BlockPlacementPolicyDefault.chooseReplicaToDelete()
Date Fri, 24 Feb 2017 23:51:44 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15883778#comment-15883778

Hudson commented on HDFS-11295:

FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11304 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11304/])
HDFS-11295. Check storage remaining instead of node remaining in (arp: rev d2b3ba9b8fb76753fa1b51661dacbde74aa5c6df)
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeStorageInfo.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java

> Check storage remaining instead of node remaining in BlockPlacementPolicyDefault.chooseReplicaToDelete()
> --------------------------------------------------------------------------------------------------------
>                 Key: HDFS-11295
>                 URL: https://issues.apache.org/jira/browse/HDFS-11295
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.7.1
>            Reporter: Xiao Liang
>            Assignee: Elek, Marton
>             Fix For: 2.9.0, 3.0.0-alpha3
>         Attachments: HDFS-11295.001.patch, HDFS-11295.002.patch, HDFS-11295.003.patch,
HDFS-11295.004.patch, HDFS-11295.jpg
> Currently in BlockPlacementPolicyDefault.chooseReplicaToDelete() the logic for choosing
replica to delete is to pick the node with the least free space(node.getRemaining()), if all
hearbeats are within the tolerable heartbeat interval.
> However, a node may have multiple storages and node.getRemaining() is a sum of the remainings
of them, if free space of the storage with the block to be delete is low, free space of the
node could still be high due to other storages of the node, finally the storage chosen may
not be the storage with least free space.
> So using storage.getRemaining() to choose a storage with least free space for choosing
replica to delete may be a better way to balance storage usage.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message