hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bharath Mundlapudi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2092) Create a light inner conf class in DFSClient
Date Fri, 24 Jun 2011 06:42:48 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054264#comment-13054264

Bharath Mundlapudi commented on HDFS-2092:

We are not concerned about the task attempt. The problem here is for Task Tracker's availability.
The way conf was designed has its own benefits. At the same time it comes with some disadvantages.
What if a task attempt can run for a day or more? This is not uncommon in, our clusters.

Again, I am listing couple of issues,
1. With UGI, conf will be created per user in TT. (Security folks?)
2. PIG or any other job can store arbitrary data. Hadoop framework should be able to deal
with it as far as it can. 
3. Last but not least, API should not hold on to client's data. 

As every job is different so can workloads can be different. So one can't see or hear all
the problems.

> Create a light inner conf class in DFSClient
> --------------------------------------------
>                 Key: HDFS-2092
>                 URL: https://issues.apache.org/jira/browse/HDFS-2092
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 0.23.0
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>             Fix For: 0.23.0
>         Attachments: HDFS-2092-1.patch, HDFS-2092-2.patch
> At present, DFSClient stores reference to configuration object. Since, these configuration
objects are pretty big at times can blot the processes which has multiple DFSClient objects
like in TaskTracker. This is an attempt to remove the reference of conf object in DFSClient.

> This patch creates a light inner conf class and copies the required keys from the Configuration

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message