hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chackaravarthy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10365) FullBlockReports retransmission delays NN startup time in large cluster.
Date Wed, 04 May 2016 19:00:15 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271208#comment-15271208
] 

Chackaravarthy commented on HDFS-10365:
---------------------------------------

Cluster details :

Version - Hadoop-2.6.0
No of datanodes - 1200
NN hardware - 74G heap allocated to NN process, 40 core machine
Total blocks - 80M+
Total Files/Directories - 60M+
Total FSObjects - 150M+
ipc.ping.interval=60s (default)
dfs.blockreport.initialDelay=120
dfs.namenode.service.handler.count=600

NN takes more than 1 minute to process FBR because of write lock getting released while processing
report for each storage. Since block report initial delay set to 120s, NN is flooded with
FBR from all DN's. After processing report for each storage, lock contention is more and by
the time it completes processing for all storages (10 data dirs configured), DN gets timeout.


What if we acquire lock at the start and release it only after processing reports for all
storages? Since FBR call frequency is very less (only during startup of NN or DN, once every
6 hours, or when a disk failure happens in DN) will this change impact the normal heartbeat/IBR
flow? Or acquiring lock at each storage report processing is done intentionally? I could not
find any comment related to this in HDFS-4987 Please correct me If I am wrong.

I could see possible config options to overcome this to increase block report initial delay
or increasing ipc.ping.interval. Also may be 600 is not correct value for service handler
count. Is there any guideline to set service handler count depending upon cluster size?

Thanks.


> FullBlockReports retransmission delays NN startup time in large cluster.
> ------------------------------------------------------------------------
>
>                 Key: HDFS-10365
>                 URL: https://issues.apache.org/jira/browse/HDFS-10365
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 2.6.0
>         Environment: version - hadoop-2.6.0
> DN - 1200 nodes
>            Reporter: Chackaravarthy
>            Priority: Critical
>
> Whenever NN is restarted, it takes huge time for NN to come back to stable state. i.e.
Last contact time remains more than 1 or 2 mins continuously for around 3 to 4 hours. This
is mainly because most of the DN's getting timeout (60s) in blockReport (FBR) rpc call and
then it keep sending FBR again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message