hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12545) Autotune NameNode RPC handler threads according to number of datanodes in cluster
Date Tue, 26 Sep 2017 15:40:02 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180960#comment-16180960
] 

Daryn Sharp commented on HDFS-12545:
------------------------------------

More threads don't always mean more throughput. Increasing the handlers increases lock contention,
context switches, gc pressure, etc. If anything, the tuning should probably be based on number
of cores.

It must also be truly configurable. The proposed formula appears to force a minimum of 200
handlers on 1k+ clusters.  We run with 100 handlers even on 5k+ clusters.  Rather than yet
another conf key, perhaps "0" should mean autotune.

> Autotune NameNode RPC handler threads according to number of datanodes in cluster
> ---------------------------------------------------------------------------------
>
>                 Key: HDFS-12545
>                 URL: https://issues.apache.org/jira/browse/HDFS-12545
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ajay Kumar
>            Assignee: Ajay Kumar
>
> Autotune NameNode RPC handler threads according to number of datanodes in cluster. Currently
rpc handler are controlled by {{dfs.namenode.handler.count}} on cluster start. Jira is to
discuss best way to auto tune it according to no of datanodes. Updating this to {{max(dfs.namenode.handler.count,
min(200,20 * log2(no of datanodes)))}} on NameNode start is one possible way. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message