hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "HuangTao (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-14419) Avoid repeated calls to the listOpenFiles function
Date Tue, 09 Apr 2019 10:19:00 GMT
HuangTao created HDFS-14419:
-------------------------------

             Summary: Avoid repeated calls to the listOpenFiles function
                 Key: HDFS-14419
                 URL: https://issues.apache.org/jira/browse/HDFS-14419
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: namenode, performance
    Affects Versions: 3.3.0
            Reporter: HuangTao
            Assignee: HuangTao
             Fix For: 3.3.0


`hdfs dfsadmin -listOpenFiles -path /any/path` will request all opened files. In the NameNode side,
the function LeaseManager.java#getUnderConstructionFiles will be called.

When there are only N(<maxListOpenFilesResponses, default is 100) files meet the conditions,
but the leaseManager contains M(>maxListOpenFilesResponses) files, we will scan all leases.
Finally, the hasMore will be set true and the openFileEntries contains N files, the scenario
will cause listOpenFiles() will be called again.

If M is greater more than N, the two calls to getUnderConstructionFiles will impact the NameNode
performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message