Mailing-List: contact pig-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: pig-dev@hadoop.apache.org
Message-ID: <591011645.1236317036208.JavaMail.jira@brutus>
Date: Thu, 5 Mar 2009 21:23:56 -0800 (PST)
From: "Gunther Hagleitner (JIRA)" <jira@apache.org>
To: pig-dev@hadoop.apache.org
Subject: [jira] Updated: (PIG-627) PERFORMANCE: multi-query optimization
In-Reply-To: <1915989148.1232498399494.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/PIG-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gunther Hagleitner updated PIG-627:
-----------------------------------

    Attachment: file_cmds-0305.patch

This patch is for the multi query branch again. It mostly fixes the problem with certain commands in the script that require immediate execution (in batch mode).

So if you do stuff like:

...
store a into 'tmp_foo';
...
rm tmp_foo
...

The rm will trigger execution and the file will be there for you to delete, copyToLocal, move, etc. You can also use the "exec" statement without params in a script now, to force execution of what we've seen so far.

This patch also contains a minor fix with the computation of progress in MR jobs (which I screwed up in the last patch).


> PERFORMANCE: multi-query optimization
> -------------------------------------
>
>                 Key: PIG-627
>                 URL: https://issues.apache.org/jira/browse/PIG-627
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: types_branch
>            Reporter: Olga Natkovich
>             Fix For: types_branch
>
>         Attachments: file_cmds-0305.patch, multi-store-0303.patch, multi-store-0304.patch, multiquery_0223.patch, multiquery_0224.patch
>
>
> Currently, if your Pig script contains multiple stores and some shared computation, Pig will execute several independent queries. For instance:
> A = load 'data' as (a, b, c);
> B = filter A by a > 5;
> store B into 'output1';
> C = group B by b;
> store C into 'output2';
> This script will result in map-only job that generated output1 followed by a map-reduce job that generated output2. As the resuld data is read, parsed and filetered twice which is unnecessary and costly. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.