hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewan Higgs (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9807) Add an optional StorageID to writes
Date Wed, 22 Feb 2017 14:58:44 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15878382#comment-15878382

Ewan Higgs commented on HDFS-9807:

The most obvious way to implement this is for the NN to choose a volume and have the DN do
what it's told. But the NN doesn't choose a precise volume. It only uses the first Storage
of a given StorageType (see {{DataNodeDescriptor.chooseStorage4Block}}). This means that all
blocks would only be written to the first volume instead of spread around.

We can either change the algorithm in {{DataNodeDescriptor}} to choose a better selection.
Or, [~chris.douglas], did you have a different policy in mind when you wrote the ticket? Perhaps
passing the Storage ID into {{FsVolumeList.getNextVolume}} and ignoring it if the {{StorageType}}
is not {{PROVIDED}}?

> Add an optional StorageID to writes
> -----------------------------------
>                 Key: HDFS-9807
>                 URL: https://issues.apache.org/jira/browse/HDFS-9807
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Chris Douglas
>            Assignee: Ewan Higgs
> The {{BlockPlacementPolicy}} considers specific storages, but when the replica is written
the DN {{VolumeChoosingPolicy}} is unaware of any preference or constraints from other policies
affecting placement. This limits heterogeneity to the declared storage types, which are treated
as fungible within the target DN. It should be possible to influence or constrain the DN policy
to select a particular storage.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message