hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dinesh Chitlangia (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-14859) Prevent Un-necessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero
Date Tue, 24 Sep 2019 02:01:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936122#comment-16936122
] 

Dinesh Chitlangia edited comment on HDFS-14859 at 9/24/19 2:00 AM:
-------------------------------------------------------------------

[~smajeti] Thanks for quick turn around. I noticed the old variable names in the comments
in BlockManagerSafeMode.java -L573- & 576 in your patch.

Either you can fix this and any other such occurrences in this file in the patch itself or
whoever is going to commit this can fix it during commit. I leave it up to you/committer :)

I usually like to fix it in patch, makes it easy for the committer.


was (Author: dineshchitlangia):
[~smajeti] Thanks for quick turn around. I noticed the old variable names in the comments
in BlockManagerSafeMode.java L573 & 576 in your patch.

Either you can fix this and any other such occurrences in this file in the patch itself or
whoever is going to commit this can fix it during commit. I leave it up to you/committer :)

I usually like to fix it in patch, makes it easy for the committer.

> Prevent Un-necessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes
is not zero
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-14859
>                 URL: https://issues.apache.org/jira/browse/HDFS-14859
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 3.1.0, 3.3.0, 3.1.4
>            Reporter: Srinivasu Majeti
>            Assignee: Srinivasu Majeti
>            Priority: Major
>              Labels: block
>         Attachments: HDFS-14859.001.patch, HDFS-14859.002.patch, HDFS-14859.003.patch
>
>
> There have been improvements like HDFS-14171 and HDFS-14632 to the performance issue
caused from getNumLiveDataNodes calls per block. The improvement has been only done w.r.t
dfs.namenode.safemode.min.datanodes paramter being set to 0 or not.
> {code}
>    private boolean areThresholdsMet() {
>      assert namesystem.hasWriteLock();
> -    int datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
> +    // Calculating the number of live datanodes is time-consuming
> +    // in large clusters. Skip it when datanodeThreshold is zero.
> +    int datanodeNum = 0;
> +    if (datanodeThreshold > 0) {
> +      datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
> +    }
>      synchronized (this) {
>        return blockSafe >= blockThreshold && datanodeNum >= datanodeThreshold;
>      }
> {code}
> I feel above logic would create similar situation of un-necessary evaluations of getNumLiveDataNodes
when dfs.namenode.safemode.min.datanodes paramter is set > 0 even though "blockSafe >=
blockThreshold" is false for most of the time in NN startup safe mode. We could do something
like below to avoid this
> {code}
> private boolean areThresholdsMet() {
>     assert namesystem.hasWriteLock();
>     synchronized (this) {
>       return blockSafe >= blockThreshold && (datanodeThreshold > 0)?
>               blockManager.getDatanodeManager().getNumLiveDataNodes() >= datanodeThreshold
: true;
>     }
>   } 
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message