hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "hemanthboyina (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-14859) Prevent Un-necessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero
Date Fri, 20 Sep 2019 06:44:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934123#comment-16934123
] 

hemanthboyina commented on HDFS-14859:
--------------------------------------

thanks for putting this up [~smajeti] 
are you working on this ?

> Prevent Un-necessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes
is not zero
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-14859
>                 URL: https://issues.apache.org/jira/browse/HDFS-14859
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 3.1.0, 3.3.0, 3.1.4
>            Reporter: Srinivasu Majeti
>            Priority: Major
>              Labels: block
>
> There have been improvements like HDFS-14171 and HDFS-14632 to the performance issue
caused from getNumLiveDataNodes calls per block. The improvement has been only done w.r.t
dfs.namenode.safemode.min.datanodes paramter being set to 0 or not.
>    private boolean areThresholdsMet() {
>      assert namesystem.hasWriteLock();
> -    int datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
> +    // Calculating the number of live datanodes is time-consuming
> +    // in large clusters. Skip it when datanodeThreshold is zero.
> +    int datanodeNum = 0;
> +    if (datanodeThreshold > 0) {
> +      datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
> +    }
>      synchronized (this) {
>        return blockSafe >= blockThreshold && datanodeNum >= datanodeThreshold;
>      }
>  
> I feel above logic would create similar situation of un-necessary evaluations of getNumLiveDataNodes
when dfs.namenode.safemode.min.datanodes paramter is set > 0 even though "blockSafe >=
blockThreshold" is false for most of the time in NN startup safe mode. We could do something
like below to avoid this
> private boolean areThresholdsMet() {
>     assert namesystem.hasWriteLock();
>     synchronized (this) {
>       return blockSafe >= blockThreshold && (datanodeThreshold > 0)?
>               blockManager.getDatanodeManager().getNumLiveDataNodes() >= datanodeThreshold
: true;
>     }
>   } 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message