hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bharath Mundlapudi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2092) Create a light inner conf class in DFSClient
Date Fri, 24 Jun 2011 05:54:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054245#comment-13054245
] 

Bharath Mundlapudi commented on HDFS-2092:
------------------------------------------

Hi Aaron,

That was just a sample of measurement for a day. We should care for MAX here in this case.
Also, Going forward, PIG 0.9 will store lots of meta data in the conf and also one can embed
the PIG script itself in the conf. This can potentially blow the TT. We can measure an approx
size of conf by the job.xml file in the job history location. Since one can store anything
in the job conf, we should be careful with the references to this object - we should not hold
for long duration. 

    

  

> Create a light inner conf class in DFSClient
> --------------------------------------------
>
>                 Key: HDFS-2092
>                 URL: https://issues.apache.org/jira/browse/HDFS-2092
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>             Fix For: 0.23.0
>
>         Attachments: HDFS-2092-1.patch, HDFS-2092-2.patch
>
>
> At present, DFSClient stores reference to configuration object. Since, these configuration
objects are pretty big at times can blot the processes which has multiple DFSClient objects
like in TaskTracker. This is an attempt to remove the reference of conf object in DFSClient.

> This patch creates a light inner conf class and copies the required keys from the Configuration
object.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message