Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Fri, 24 Jun 2011 17:59:50 +0000 (UTC)
From: "Aaron T. Myers (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: 
 <338559931.37836.1308938390302.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1271780704.23333.1308632387413.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HDFS-2092) Create a light inner conf class in
 DFSClient
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054585#comment-13054585 ] 

Aaron T. Myers commented on HDFS-2092:
--------------------------------------

bq. Hi Aaron, we did see some cases in the past that some users put a large object in conf and then JT/TT ran out of memory. Indeed, users can put arbitrary large objects in conf.

Thanks for this explanation, Nicholas. That does indeed seem like a problem worthy of attack.

bq. So this change also prevents such problems.

I'm not entirely convinced of this. Does this change definitely prevent these problems? Is it really the case that the JT could've garbage collected these {{JobConf}} instances, were it not for the {{DFSClient}} still holding a reference? If that's the intended goal, I'd really like to see a little benchmark done demonstrating the memory use of the JT with large {{JobConf}} objects before and after this patch. If this patch does indeed address this issue, I could even imagine a unit test being written which could ensure that no long-lived {{JobConf}} references sneak back into the JT.

> Create a light inner conf class in DFSClient
> --------------------------------------------
>
>                 Key: HDFS-2092
>                 URL: https://issues.apache.org/jira/browse/HDFS-2092
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>             Fix For: 0.23.0
>
>         Attachments: HDFS-2092-1.patch, HDFS-2092-2.patch
>
>
> At present, DFSClient stores reference to configuration object. Since, these configuration objects are pretty big at times can blot the processes which has multiple DFSClient objects like in TaskTracker. This is an attempt to remove the reference of conf object in DFSClient. 
> This patch creates a light inner conf class and copies the required keys from the Configuration object.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira