hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Dai <jiany...@yahoo-inc.com>
Subject Re: split operator
Date Mon, 26 Jul 2010 20:58:49 GMT
Hi, Gang,
It is about multiquery optimization. In MRCompiler, we will create a
map-reduce boundary for split, later in MultiQueryOptimizer, we will
merge several split into one map-reduce job. In this map-reduce job, we
will nest several split plans.

Daniel

Gang Luo wrote:
> Hi Daniel,
> in 4.3.1, the example and figure 6 show this. 5.1 last paragraph says split 
> operator maintain one-tuple buffer for each branch and talks about how to 
> synchronize multiple branches. I do think that is the in-memory split.
>
> here is the paper: http://www.vldb.org/pvldb/2/vldb09-1074.pdf
>
>
> -Gang
>
>
>
> ----- 原始邮件 ----
> 发件人: Daniel Dai <jianyong@yahoo-inc.com>
> 收件人: "pig-dev@hadoop.apache.org" <pig-dev@hadoop.apache.org>
> 发送日期: 2010/7/26 (周一) 2:09:25 下午
> 主   题: Re: split operator
>
> Hi, Gang,
> Which part of the paper are you talking about? We don't do in-memory split. We 
> dump the split result to a temporary file and start a new map-reduce job. Split 
> do create a map-reduce boundary (Though it is not entirely true, multiquery 
> optimizer may combine some of these jobs)
>
> Daniel
>
> Gang Luo wrote:
>   
>> Hi all
>> according to the vldb 09 paper, the split operator and all its successive 
>> operators reside in memory without any blocking in between. However, the source 
>> code (version 0.7) shows that a MR job is actually ended when it meets the split

>> operator and multiple new MR jobs are created, each representing one branch. 
>> This write-once-read-multiple-times method is different from the in-memory 
>> method mentioned in that paper. Does pig change the strategy for split, or is 
>> there still an in-memory version of split I didn't discover?
>>
>> Thanks,
>> -Gang
>>
>>
>>        
>>     
>
>
>       
>   


Mime
View raw message