hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11907) NameNodeResourceChecker should avoid calling df.getAvailable too frequently
Date Thu, 08 Jun 2017 21:14:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043450#comment-16043450
] 

Andrew Wang commented on HDFS-11907:
------------------------------------

I would appreciate some diligence investigating the root cause of the problem. I don't think
we should commit a behavior change when there isn't a degree of confidence that it'll solve
the stated problem.

Based on the discussion, we (Kihwal, myself, Arpit) expect {{df}} to be a cheap call. Given
that, there is not a degree of confidence that this change will speedup health checks. I'd
like to hear at least a theory as to why {{df}} would be slow in a spurious way before we
add complexity.

I'm not going to burn bridges over this patch if someone *really* wants to commit it, but
as I said before, I would much rather see a patch to add new metrics or instrumentation to
root cause the problem.

> NameNodeResourceChecker should avoid calling df.getAvailable too frequently
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-11907
>                 URL: https://issues.apache.org/jira/browse/HDFS-11907
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Chen Liang
>            Assignee: Chen Liang
>         Attachments: HDFS-11907.001.patch, HDFS-11907.002.patch, HDFS-11907.003.patch,
HDFS-11907.004.patch
>
>
> Currently, {{HealthMonitor#doHealthChecks}} invokes {{NameNode#monitorHealth}} which
ends up invoking {{NameNodeResourceChecker#isResourceAvailable}}, at the frequency of once
per second by default. And NameNodeResourceChecker#isResourceAvailable invokes {{df.getAvailable();}}
every time it is called.
> Since available space information should rarely be changing dramatically at the pace
of per second. A cached value should be sufficient. i.e. only try to get the updated value
when the cached value is too old. otherwise simply return the cached value. This way df.getAvailable()
gets invoked less.
> Thanks [~arpitagarwal] for the offline discussion.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message