hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Ochoa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7787) Split QUEUE_HIGHEST_PRIORITY in UnderReplicatedBlocks to give more priority to blocks on nodes being decomissioned
Date Fri, 03 Mar 2017 08:48:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893930#comment-15893930
] 

Daniel Ochoa commented on HDFS-7787:
------------------------------------

I'm having the same issue (decommission process is taking too long), blocks with no live replicas
except on nodes that decommissioning should have highest priority.

The code is now here:
https://github.com/apache/hadoop/blob/b61fb267b92b2736920b4bd0c673d31e7632ebb9/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/LowRedundancyBlocks.java

> Split QUEUE_HIGHEST_PRIORITY in UnderReplicatedBlocks to give more priority to blocks
on nodes being decomissioned
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-7787
>                 URL: https://issues.apache.org/jira/browse/HDFS-7787
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.6.0
>         Environment: 2 namenodes HA, 6 datanodes in two racks
>            Reporter: Frode Halvorsen
>              Labels: balance, hdfs, replication-performance
>
> Each file has a setting of 3 replicas. split on different racks.
> After a simulated crash of one rack (shutdown of all nodes, deleted data-directory an
started nodes) and decommssion of one of the nodes in the orther rack the replication does
not follow 'normal' rules...
> My cluster has appx 25 mill files, and the one node I now try to decommision has 9 millions
underreplicated blocks, and 3,5 million blocks with 'no live replicas'. After a restart of
the node, it starts to replicate both types of blocks, but after a while, it only repliates
under-replicated blocks with other live copies. I would think that the 'normal' way to do
this would be to make sure that all blocks this node keeps the only copy of, should be the
first to be replicated/balanced ?  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message