pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gunther Hagleitner (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-627) PERFORMANCE: multi-query optimization
Date Fri, 06 Mar 2009 05:23:56 GMT

     [ https://issues.apache.org/jira/browse/PIG-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Gunther Hagleitner updated PIG-627:
-----------------------------------

    Attachment: file_cmds-0305.patch

This patch is for the multi query branch again. It mostly fixes the problem with certain commands
in the script that require immediate execution (in batch mode).

So if you do stuff like:

...
store a into 'tmp_foo';
...
rm tmp_foo
...

The rm will trigger execution and the file will be there for you to delete, copyToLocal, move,
etc. You can also use the "exec" statement without params in a script now, to force execution
of what we've seen so far.

This patch also contains a minor fix with the computation of progress in MR jobs (which I
screwed up in the last patch).



> PERFORMANCE: multi-query optimization
> -------------------------------------
>
>                 Key: PIG-627
>                 URL: https://issues.apache.org/jira/browse/PIG-627
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>             Fix For: types_branch
>
>         Attachments: file_cmds-0305.patch, multi-store-0303.patch, multi-store-0304.patch,
multiquery_0223.patch, multiquery_0224.patch
>
>
> Currently, if your Pig script contains multiple stores and some shared computation, Pig
will execute several independent queries. For instance:
> A = load 'data' as (a, b, c);
> B = filter A by a > 5;
> store B into 'output1';
> C = group B by b;
> store C into 'output2';
> This script will result in map-only job that generated output1 followed by a map-reduce
job that generated output2. As the resuld data is read, parsed and filetered twice which is
unnecessary and costly. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message