hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod K V (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1201) [zebra] HDFS meta queries are issued by all mappers; Pig Loader serialize all JobConf contents including those unused by zebra
Date Mon, 25 Jan 2010 08:29:34 GMT

    [ https://issues.apache.org/jira/browse/PIG-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804443#action_12804443
] 

Vinod K V commented on PIG-1201:
--------------------------------

I am not at all familiar with zebra or pig code, but just checked the patch to verify the
JobConf issue.
{code}
-			Configuration localConf = new Configuration();
+			Configuration localConf = ConfigurationUtil.toConfiguration(store.getConfiguration());
{code}
{{ConfigurationUtil.toConfiguration()}} still uses a {{new Configuration()}} which will initialize
a new object with *all* the default properties from the default resource(*-default.xml)s also
populated. So the size of the config object will still be huge, or atleast not small. Instead,
I think we should use {{new Configuration(false)}}.

Am I making any sense or are my observations completely invalid?

> [zebra] HDFS meta queries are issued by all mappers; Pig Loader serialize all JobConf
contents including those unused by zebra
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PIG-1201
>                 URL: https://issues.apache.org/jira/browse/PIG-1201
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.6.0
>            Reporter: Yan Zhou
>            Assignee: Yan Zhou
>            Priority: Minor
>             Fix For: 0.6.0, 0.7.0
>
>         Attachments: PIG-1201.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message