pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gang Luo <lgpub...@yahoo.com.cn>
Subject Re: split operator
Date Mon, 23 Aug 2010 23:20:39 GMT
Hi Daniel,
This is a question long ago, but I suddenly come up with some more thoughts on 
this. In a query as simple as this:

A = LOAD 'input';
B = FILTER A BY $1 == 1;
C = COGROUP A BY $0, B BY $0;

the optimizer will insert a split operator to reuse A. According to the source 
code, a map-reduce job will be ended when it sees split and output the result to 
A1 and A2 which will be used by two subsequent jobs to process B and C. In this 
case, the first job does nothing meaningful but copy the souce 'input' twice. Is 
there some optimization applied here (like the MultiQueryOptimizer you mentioned 
previously) ? How?

Since I didn't take a look at the MultiQueryOptimizer, it will be great help if 
you can briefly describe how MultiQueryOptimizer works. Thanks a lot.


----- 原始邮件 ----
发件人: Daniel Dai <jianyong@yahoo-inc.com>
收件人: "pig-dev@hadoop.apache.org" <pig-dev@hadoop.apache.org>
发送日期: 2010/7/26 (周一) 4:58:49 下午
主   题: Re: split operator

Hi, Gang,
It is about multiquery optimization. In MRCompiler, we will create a
map-reduce boundary for split, later in MultiQueryOptimizer, we will
merge several split into one map-reduce job. In this map-reduce job, we
will nest several split plans.


Gang Luo wrote:
> Hi Daniel,
> in 4.3.1, the example and figure 6 show this. 5.1 last paragraph says split 
> operator maintain one-tuple buffer for each branch and talks about how to 
> synchronize multiple branches. I do think that is the in-memory split.
> here is the paper: http://www.vldb.org/pvldb/2/vldb09-1074.pdf
> -Gang
> ----- 原始邮件 ----
> 发件人: Daniel Dai <jianyong@yahoo-inc.com>
> 收件人: "pig-dev@hadoop.apache.org" <pig-dev@hadoop.apache.org>
> 发送日期: 2010/7/26 (周一) 2:09:25 下午
> 主   题: Re: split operator
> Hi, Gang,
> Which part of the paper are you talking about? We don't do in-memory split. We 

> dump the split result to a temporary file and start a new map-reduce job. Split 
> do create a map-reduce boundary (Though it is not entirely true, multiquery 
> optimizer may combine some of these jobs)
> Daniel
> Gang Luo wrote:
>> Hi all
>> according to the vldb 09 paper, the split operator and all its successive 
>> operators reside in memory without any blocking in between. However, the source 
>> code (version 0.7) shows that a MR job is actually ended when it meets the 
>> operator and multiple new MR jobs are created, each representing one branch. 
>> This write-once-read-multiple-times method is different from the in-memory 
>> method mentioned in that paper. Does pig change the strategy for split, or is 

>> there still an in-memory version of split I didn't discover?
>> Thanks,
>> -Gang


View raw message