hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yiqun Lin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11464) Improve the selection in choosing storage for blocks
Date Fri, 03 Mar 2017 12:32:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894253#comment-15894253

Yiqun Lin commented on HDFS-11464:

Thanks [~arpitagarwal] for the comment. You are right, datanode only cares the storage type
here. This was implemented by HDFS-8946 and giving the detail reason of current logic there.
Linked to HDFS-8946.

> Improve the selection in choosing storage for blocks
> ----------------------------------------------------
>                 Key: HDFS-11464
>                 URL: https://issues.apache.org/jira/browse/HDFS-11464
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Yiqun Lin
>            Assignee: Yiqun Lin
>         Attachments: HDFS-11464.001.patch
> Currently the logic in choosing storage for blocks is not a good way. It always uses
the first valid storage of a given StorageType ({{see DataNodeDescriptor#chooseStorage4Block}}).
This should not be a good selection. That means blcoks will always be written to the same
volume (first volume) and other valid volumes have no choices. This problem is brought up
by this comment ( https://issues.apache.org/jira/browse/HDFS-9807?focusedCommentId=15878382&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15878382
> There is one solution from me:
> * First, based on existing storages in one node, extract all the valid storages into
a collection.
> * Then, disrupt the order of these vaild storages, get a new collection.
> * Finally, get the first storage from the new storages collection.
> These steps will be executed in {{DataNodeDescriptor#chooseStorage4Block}} and replace
current logic. I think this improvement can be done as a subtask under HDFS-11419. Any further
comments are welcomed.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message