hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rakesh R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10800) [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks which were placed in other storages than what NN is expecting.
Date Wed, 21 Sep 2016 07:22:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509035#comment-15509035

Rakesh R commented on HDFS-10800:

Thanks [~umamaheswararao], latest patch looks good. Few more suggestions to improve the patch.
sorry, I failed to identify this in my previous review, would you mind incorporating this.
# We could mark the new classes {{@InterfaceAudience.Private}}
# Can we find a better name for ‘StorageMovementNeeded’ class, this name looks like its
moving the storage. How about BlockMovementNeeded or BlockStorageMovementNeeded?
# It would be good if we could {{sps.stop();}} the thread at the beginning of the {{BlockManager#stop()}}
function. This could avoid unwanted exceptions in future if someone add extra codes which
depends on already stopped dependent services
# IMHO, {{#assignStorageMovementTasksToDN()}} can be moved inside {{#buildStorageMismatchedBlocks(long
blockCollectionID)}}, that way we could avoid passing few args. Also, the below if condition
is not required as duplicate checks inside the assign function, right?
if (blockInfoToMoveStorages.size() > 0) {
                  assignStorageMovementTasksToDN(coordinatorNode, blockCollectionID,
# Also, can you make the following variable as {{final}}. This will avoid NPE if someone mistakenly
do any changes in future.
private StoragePolicySatisfier sps;
private StorageMovementNeededneedStorageMovement = newStorageMovementNeeded();

> [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks which were
placed in other storages than what NN is expecting.
> -------------------------------------------------------------------------------------------------------------------------------------------
>                 Key: HDFS-10800
>                 URL: https://issues.apache.org/jira/browse/HDFS-10800
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: HDFS-10285
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>         Attachments: HDFS-10800-HDFS-10285-00.patch, HDFS-10800-HDFS-10285-01.patch,
HDFS-10800-HDFS-10285-02.patch, HDFS-10800-HDFS-10285-03.patch
> This JIRA is for implementing a daemon thread called StoragePolicySatisfier in nematode,
which should scan the asked files blocks which were placed in wrong storages in DNs. 
>  The idea is:
>       # When user called on some files/dirs for satisfyStorage policy, They should have
tracked in NN and then StoragePolicyDaemon thread will pick one by one file and then check
the blocks which might have placed in wrong storage in DN than what NN is expecting it to.
>       # After checking all, it should also construct the data structures for the required
information to move a block from one storage to another.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message