hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gunther Hagleitner (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-777) Code refactoring: Create optimization out of store/load post processing code
Date Wed, 29 Apr 2009 06:15:31 GMT

    [ https://issues.apache.org/jira/browse/PIG-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704002#action_12704002

Gunther Hagleitner commented on PIG-777:


Per PIG-627 the first example you gave will result in a single map reduce job that is going
to process both store operations. No duplication of steps A thru D. So, yes you shouldn't
need to introduce "D = load". Also PIG-627 introduced an optimization that will throw the
"D = load" out - basically transforming your second example into the first.

This bug is mostly about the way the optimization is written. Some code should be moved around
to align it with the optimization framework.

Adding a log message when this happens is a good idea though. Let me add that. 

> Code refactoring: Create optimization out of store/load post processing code
> ----------------------------------------------------------------------------
>                 Key: PIG-777
>                 URL: https://issues.apache.org/jira/browse/PIG-777
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Gunther Hagleitner
> The postProcessing method in the pig server checks whether a logical graph contains stores
to and loads from the same location. If so, it will either connect the store and load, or
optimize by throwing out the load and connecting the store predecessor with the successor
of the load.
> Ideally the introduction of the store and load connection should happen in the query
compiler, while the optimization should then happen in an separate optimizer step as part
of the optimizer framework.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message