hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13234) Remove renew configuration instance in ConfiguredFailoverProxyProvider and reduce memory footprint for client
Date Wed, 07 Mar 2018 20:24:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390147#comment-16390147
] 

Kihwal Lee commented on HDFS-13234:
-----------------------------------

bq. Configuration occupies over 600MB
How big is a single instance in your use case? Bloated conf in dfs client is obviously a serious
issue, but it can create bigger issues in apps/jobs. Sometimes a conf can get embedded in
another conf.  Avoiding unnecessarily duplicated confs is a good thing, but looking into what
is causing the bloat and fixing that will also be important.

> Remove renew configuration instance in ConfiguredFailoverProxyProvider and reduce memory
footprint for client
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-13234
>                 URL: https://issues.apache.org/jira/browse/HDFS-13234
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: fs, ha, hdfs-client
>            Reporter: He Xiaoqiao
>            Priority: Major
>         Attachments: HDFS-13234.001.patch
>
>
> The memory footprint of #DFSClient is very considerable in some special scenario since
there are many #Configuration instances and occupy much memory resource (In an extreme case,
org.apache.hadoop.conf.Configuration occupies over 600MB we meet under HDFS Federation an
HA with QJM and there are dozens of NameNodes). I think some new Configuration instance is
not necessary. Such as  #ConfiguredFailoverProxyProvider initialization.
> {code:java}
>   public ConfiguredFailoverProxyProvider(Configuration conf, URI uri,
>       Class<T> xface, HAProxyFactory<T> factory) {
>     this.xface = xface;
>     this.conf = new Configuration(conf);
>     ......
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message