hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "HuangTao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-14419) Avoid repeated calls to the listOpenFiles function
Date Tue, 09 Apr 2019 10:22:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-14419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

HuangTao updated HDFS-14419:
    Attachment: HDFS-14419.001.patch

> Avoid repeated calls to the listOpenFiles function
> --------------------------------------------------
>                 Key: HDFS-14419
>                 URL: https://issues.apache.org/jira/browse/HDFS-14419
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode, performance
>    Affects Versions: 3.3.0
>            Reporter: HuangTao
>            Assignee: HuangTao
>            Priority: Minor
>             Fix For: 3.3.0
>         Attachments: HDFS-14419.001.patch
> `hdfs dfsadmin -listOpenFiles -path /any/path` will request all opened files. In the NameNode side,
the function LeaseManager.java#getUnderConstructionFiles will be called.
> When there are only N(<maxListOpenFilesResponses, default is 100) files meet the conditions,
but the leaseManager contains M(>maxListOpenFilesResponses) files, we will scan all leases.
Finally, the hasMore will be set true and the openFileEntries contains N files, the scenario
will cause listOpenFiles() will be called again.
> If M is greater more than N, the two calls to getUnderConstructionFiles will impact
the NameNode performance.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message