hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8789) Block Placement policy migrator
Date Wed, 10 May 2017 14:47:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004804#comment-16004804

Kihwal Lee commented on HDFS-8789:

What happens on NN restart (to be precise, during the replication queues initialization)?

Each block will be checked against {{BlockPlacementPolicy#verifyBlockPlacement()}} and will
get added to the replication queue. When calculating repl work, {{chooseTarget()}} is supposed
to help correct any violation. This happens when the network topology or the placement policy
changes.  Does it also work for upgrade domain block placement policy? It will stress namenode
if there are a lot of replicas to be migrated, but will do the job.  I.e. the option number
2 might be already there.

If we are worried about overwhelming NN, we could have a separate thread to do throttled scan
of block placement policy violations. The replication candidate generation rate would also
be throttled. The repl queue init would only deal with truly under-replicated cases. 

> Block Placement policy migrator
> -------------------------------
>                 Key: HDFS-8789
>                 URL: https://issues.apache.org/jira/browse/HDFS-8789
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Chris Trezzo
>            Assignee: Chris Trezzo
>         Attachments: HDFS-8789-trunk-STRAWMAN-v1.patch
> As we start to add new block placement policies to HDFS, it will be necessary to have
a robust tool that can migrate HDFS blocks between placement policies. This jira is for the
design and implementation of that tool.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message