hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6425) Large postponedMisreplicatedBlocks has impact on blockReport latency
Date Thu, 07 Aug 2014 22:35:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089967#comment-14089967
] 

Ming Ma commented on HDFS-6425:
-------------------------------

Thanks, Arpit. This jira can address more common NN failover scenario with lots of "content
stale" storages.

We try to get storages out of "content stale"  as soon as possible. Here are several scenarios.

a. For non-HA NN restart, have DN send HB before BR right after registration.
b. For HA setup, NN becomes active right after it restarts. This can happen if we have to
restart both NNs at the same time, due to some rare outage or some incompatible upgrade. In
this case, the active NN will first go to standby, then get transitioned to active at which
point all DNs will be marked as stale again. For big clusters, most of the DN reregistration
will come in after the NN becomes active, so the fix to have DNs send HB and BR right after
registration will also help.
c. For HA setup, NN becomes active after the NN JVM has been up for some time. The failover
could happen due to zk session timeout, or the other NN just crashes. In this case, there
is no DN reregistration given the new active NN doesn't have recent restart. We can change
the NN to ask DN to resend blockreport upon failover, but that will cause cluster performance
issue.

So we still have some scenario where we might have lots of "content stale" storages. This
jira tries to make NN handle the scenario better.


> Large postponedMisreplicatedBlocks has impact on blockReport latency
> --------------------------------------------------------------------
>
>                 Key: HDFS-6425
>                 URL: https://issues.apache.org/jira/browse/HDFS-6425
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Ming Ma
>            Assignee: Ming Ma
>         Attachments: HDFS-6425-Test-Case.pdf, HDFS-6425.patch
>
>
> Sometimes we have large number of over replicates when NN fails over. When the new active
NN took over, over replicated blocks will be put to postponedMisreplicatedBlocks until all
DNs for that block aren't stale anymore.
> We have a case where NNs flip flop. Before postponedMisreplicatedBlocks became empty,
NN fail over again and again. So postponedMisreplicatedBlocks just kept increasing until the
cluster is stable. 
> In addition, large postponedMisreplicatedBlocks could make rescanPostponedMisreplicatedBlocks
slow. rescanPostponedMisreplicatedBlocks takes write lock. So it could slow down the block
report processing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message