hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Virajith Jalaparti (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-9807) Add an optional StorageID to writes
Date Thu, 04 May 2017 19:19:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997232#comment-15997232
] 

Virajith Jalaparti edited comment on HDFS-9807 at 5/4/17 7:19 PM:
------------------------------------------------------------------

Posted a new patch (v010) that reverts the unrelated changes to {{Host2NodesMap}} and {{BlockPlacementPolicyDefault}}
as [~chris.douglas] pointed out. [~ehiggs], are you ok with this? The changes to those classes
in the earlier patches don't seem needed. 

[~linyiqun] I think implementing a new {{VolumeChoosingPolicy}} will be one way in which the
Namenode's choice of the {{storageID}} is respected. As in the comments above ([this | https://issues.apache.org/jira/browse/HDFS-9807?focusedCommentId=15930679&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15930679]
and [this | https://issues.apache.org/jira/browse/HDFS-9807?focusedCommentId=15929230&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15929230]),
it will be up to the {{VolumeChoosingPolicy}} to determine if the {{storageID}} from the Namenode
is used. For the choice of {{storageID}} to be really respected, I think the {{BlockPlacementPolicy}}
on the Namenode and the {{VolumeChoosingPolicy}} should be consistent in the way the volumes
are chosen. The goal of this JIRA was to only provide the plumbing needed to propagate the
{{storageID}} to the {{VolumeChoosingPolicy}} and not to implement a new {{VolumeChoosingPolicy}}.
The actual policies to use can be determined separately.
The proposal in  HDFS-11464 will make sense. However, I think the change must be done in {{BlockPlacementPolicyDefault#chooseStorage4Block}}
and not in {{DataNodeDescriptor#chooseStorage4Block}}. Looking at {{DataNodeDescriptor#chooseStorage4Block}}
now, I think the logic implemented in {{DataNodeDescriptor#chooseStorage4Block}} can be done
in {{BlockPlacementPolicyDefault#chooseStorage4Block}} itself.


was (Author: virajith):
Posted a new patch (v010) that reverts the unrelated changes to {{Host2NodesMap}} and {{BlockPlacementPolicyDefault}}
as [~chris.douglas] pointed out. [~ehiggs], are you ok with this? The changes to those classes
in the earlier patches don't seem needed. 

[~linyiqun] I think implementing a new {{VolumeChoosingPolicy}} will be one way in which the
Namenode's choice of the {{storageID}} is respected. As in the comments above ([this | https://issues.apache.org/jira/browse/HDFS-9807?focusedCommentId=15930679&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15930679]
and [this | https://issues.apache.org/jira/browse/HDFS-9807?focusedCommentId=15929230&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15929230]),
it will be up to the {{VolumeChoosingPolicy}} to determine if the {{storageID}} from the Namenode
is used. For the choice of {{storageID}} to be really respected, I think the {{BlockPlacementPolicy}}
on the Namenode and the {{VolumeChoosingPolicy}} should be consistent in the way the volumes
are chosen. The goal of this JIRA was to only provide the plumbing needed to propagate the
{{storageID}} to the {{VolumeChoosingPolicy}} and not to implement a new {{VolumeChoosingPolicy}}.
The actual policies to use can be determined separately.

> Add an optional StorageID to writes
> -----------------------------------
>
>                 Key: HDFS-9807
>                 URL: https://issues.apache.org/jira/browse/HDFS-9807
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Chris Douglas
>            Assignee: Ewan Higgs
>         Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch, HDFS-9807.003.patch, HDFS-9807.004.patch,
HDFS-9807.005.patch, HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch, HDFS-9807.009.patch,
HDFS-9807.010.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the replica is written
the DN {{VolumeChoosingPolicy}} is unaware of any preference or constraints from other policies
affecting placement. This limits heterogeneity to the declared storage types, which are treated
as fungible within the target DN. It should be possible to influence or constrain the DN policy
to select a particular storage.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message