hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brahma Reddy Battula (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12128) Namenode failover may make balancer's efforts be in vain
Date Thu, 13 Jul 2017 01:44:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085029#comment-16085029

Brahma Reddy Battula commented on HDFS-12128:

HDFS-11146 might help on this..? Even I considered this scenario.

> Namenode failover may make balancer's efforts be in vain
> --------------------------------------------------------
>                 Key: HDFS-12128
>                 URL: https://issues.apache.org/jira/browse/HDFS-12128
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: balancer & mover
>    Affects Versions: 2.6.0
>            Reporter: liuyiyang
> The problem can be reproduced as follows:
> 1.In an HA cluster with imbalance datanode usage, we run "start-balancer.sh" to make
the cluster balanced;
> 2.Before starting balancer, trigger failover of namenodes, this will make all datanodes
be marked as stale by active namenode;
> 3.Start balancer to make the datanode usage balanced;
> 4.As balancer is running, under-utilized datanodes' usage will increase, but over-utilized
datanodes' usage will stay unchanged for long time.
> Since all datanodes are marked as stale, deletion will be postponed in stale datanodes.
During balancing, the replicas in source datanodes can't be deleted immediately,
> so the total usage of the cluster will increase and won't decrease until datanodes' stale
state be cancelled.
> When the datanodes send next block report to namenode(default interval is 6h), active
namenode will cancel the stale state of datanodes. I found if replicas on source datanodes
can't be deleted immediately in OP_REPLACE operation via del_hint to namenode,
> namenode will schedule replicas on datanodes with least remaining space to delete instead
of replicas on source datanodes. Unfortunately, datanodes with least remaining space may be
the target datanodes when balancing, which will lead to imbalanced datanode usage again.
> If balancer finishes before next block report, all postponed over-replicated replicas
will be deleted based on remaining space of datanodes, this may lead to furitless balancer

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message