hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chackravarthy Esakkimuthu <chaku.mi...@gmail.com>
Subject Guideline on setting Namenode RPC Handler count (client and service)
Date Fri, 29 Apr 2016 11:30:21 GMT
Hi,

Is there any recommendation or guideline on setting no of RPC handlers in
Namenode based on cluster size (no of datanodes)?

Cluster details :

No of datanodes - 1200
NN hardware - 74G heap allocated to NN process, 40 core machine
Total blocks - 80M+
Total Files/Directories - 60M+
Total FSObjects - 150M+

We have isolated service and client RPC by enabling service-rpc.

Currently dfs.namenode.handler.count=400 and
dfs.namenode.service.handler.count=200

Is 200 good fit for this cluster or any change recommended. Please help out.

Thanks in advance!

(We have tried increasing service handler count to 600 and have seen delay
in NN startup time and then it looked quite stable. And setting it to 200
decreases the delay in startup time but it has slightly higher rpcQueueTime
and rpcAvgProcessingTime comparing to 600 handler count.)

Thanks,
Chackra

Mime
View raw message