hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "He Xiaoqiao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-14576) Avoid block report retry and slow down namenode startup
Date Mon, 17 Jun 2019 15:55:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16865718#comment-16865718
] 

He Xiaoqiao commented on HDFS-14576:
------------------------------------

Thanks [~jojochuang],[~dineshchitlangia] for your quick response.
{quote}I suppose this can still be an issue at 10k node scale.{quote}
Correct, I think it is more serious for larger cluster according to my experience. I will
submit draft patch later and welcome discussion.
About load fsimage, there are some solution proposed, for instance HDFS-7784 loading in parallel.
I think it is time to reevaluate this solution. FYI.

> Avoid block report retry and slow down namenode startup
> -------------------------------------------------------
>
>                 Key: HDFS-14576
>                 URL: https://issues.apache.org/jira/browse/HDFS-14576
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: He Xiaoqiao
>            Assignee: He Xiaoqiao
>            Priority: Major
>
> During namenode startup, the load will be very high since it has to process every datanodes
blockreport one by one. If there are hundreds datanodes block reports pending process, the
issue will be more serious even #processFirstBlockReport is processed a lot more efficiently
than ordinary block reports. Then some of datanode will retry blockreport and lengthens restart
times. I think we should filter the block report request (via datanode blockreport retries)
which has be processed and return directly then shorten down restart time. I want to state
this proposal may be obvious only for large cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message