hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Íñigo Goiri (JIRA) <j...@apache.org>
Subject [jira] [Commented] (HDFS-14728) RBF:GetDatanodeReport causes a large GC pressure on the NameNodes
Date Tue, 13 Aug 2019 16:51:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906396#comment-16906396

Íñigo Goiri commented on HDFS-14728:

A few comments:
* I think the unit tests are having some issues.
* I would not change the default value from 10 seconds to 10 minutes. Changing defaults is
always tricky.
* In RouterRpcServer#917, it should initialize the full trace to {{throw new IOException(throwable);}}
or similar. By the way, we should cover what this exception is and have tests if so.
* In the asserts, you should put the expected value first.
* It would be good to add comments through testDatanodeReportCache explaining what are the
reasons for that sequence. Right now it is easy to see that it tries to verify that it does
not go to the NN but this may not be as clear in a few months.

> RBF:GetDatanodeReport causes a large GC pressure on the NameNodes
> -----------------------------------------------------------------
>                 Key: HDFS-14728
>                 URL: https://issues.apache.org/jira/browse/HDFS-14728
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: rbf
>            Reporter: xuzq
>            Assignee: xuzq
>            Priority: Major
>         Attachments: HDFS-14728-trunk-001.patch
> When a cluster contains millions of DNs, *GetDatanodeReport* is pretty expensive, and
it will cause a large GC pressure on NameNode.
> When multiple NSs share the millions DNs by federation and the router listens to the
NSs, the problem will be more serious.
> All the NSs will be GC at the same time.
> RBF should cache the datanode report informations and have an option to disable the cache.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message