hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11464) Improve the selection in choosing storage for blocks
Date Fri, 10 Mar 2017 01:42:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904232#comment-15904232

Chris Douglas commented on HDFS-11464:

bq. the storage chosen by the NameNode doesn't matter as the DataNode picks target volumes
(via VolumeChoosingPolicy)

The storage chosen by the NameNode could be included and passed to the DataNode (HDFS-9807).
We need to pass this information for tiered storage, but it could also be used for centralized

> Improve the selection in choosing storage for blocks
> ----------------------------------------------------
>                 Key: HDFS-11464
>                 URL: https://issues.apache.org/jira/browse/HDFS-11464
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Yiqun Lin
>            Assignee: Yiqun Lin
>         Attachments: HDFS-11464.001.patch
> Currently the logic in choosing storage for blocks is not a good way. It always uses
the first valid storage of a given StorageType ({{see DataNodeDescriptor#chooseStorage4Block}}).
This should not be a good selection. That means blcoks will always be written to the same
volume (first volume) and other valid volumes have no choices. This problem is brought up
by this comment ( https://issues.apache.org/jira/browse/HDFS-9807?focusedCommentId=15878382&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15878382
> There is one solution from me:
> * First, based on existing storages in one node, extract all the valid storages into
a collection.
> * Then, disrupt the order of these vaild storages, get a new collection.
> * Finally, get the first storage from the new storages collection.
> These steps will be executed in {{DataNodeDescriptor#chooseStorage4Block}} and replace
current logic. I think this improvement can be done as a subtask under HDFS-11419. Any further
comments are welcomed.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message