hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Virajith Jalaparti (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9807) Add an optional StorageID to writes
Date Thu, 04 May 2017 18:59:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997232#comment-15997232
] 

Virajith Jalaparti commented on HDFS-9807:
------------------------------------------

Posted a new patch (v010) that reverts the unrelated changes to {{Host2NodesMap}} and {{BlockPlacementPolicyDefault}}
as [~chris.douglas] pointed out. [~ehiggs], are you ok with this? The changes to those classes
in the earlier patches don't seem needed. 

[~linyiqun] I think implementing a new {{VolumeChoosingPolicy}} will be one way in which the
Namenode's choice of the {{storageID}} is respected. As in the comments above ([this | https://issues.apache.org/jira/browse/HDFS-9807?focusedCommentId=15930679&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15930679]
and [this | https://issues.apache.org/jira/browse/HDFS-9807?focusedCommentId=15929230&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15929230]),
it will be up to the {{VolumeChoosingPolicy}} to determine if the {{storageID}} from the Namenode
is used. For the choice of {{storageID}} to be really respected, I think the {{BlockPlacementPolicy}}
on the Namenode and the {{VolumeChoosingPolicy}} should be consistent in the way the volumes
are chosen. The goal of this JIRA was to only provide the plumbing needed to propagate the
{{storageID}} to the {{VolumeChoosingPolicy}} and not to implement a new {{VolumeChoosingPolicy}}.
The actual policies to use can be determined separately.

> Add an optional StorageID to writes
> -----------------------------------
>
>                 Key: HDFS-9807
>                 URL: https://issues.apache.org/jira/browse/HDFS-9807
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Chris Douglas
>            Assignee: Ewan Higgs
>         Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch, HDFS-9807.003.patch, HDFS-9807.004.patch,
HDFS-9807.005.patch, HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch, HDFS-9807.009.patch,
HDFS-9807.010.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the replica is written
the DN {{VolumeChoosingPolicy}} is unaware of any preference or constraints from other policies
affecting placement. This limits heterogeneity to the declared storage types, which are treated
as fungible within the target DN. It should be possible to influence or constrain the DN policy
to select a particular storage.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message