hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renato Marroquín Mogrovejo <renatoj.marroq...@gmail.com>
Subject Re: Pig optimizer
Date Sat, 28 Aug 2010 19:48:56 GMT
Hi Daniel,

Yeah that is, but like there are two types of optimizations right? I mean
physical and logical optimizations. The physical ones are the ones on how
the operators are distributed along mapreduce jobs and the logical ones are
the ones similar to relational algebra right?
Do you have any tips on how to get a quick grasp on pig logical
optimizations?
Thanks again.


Renato M.


2010/8/26 Daniel Dai <jianyong@yahoo-inc.com>

> Hi, Renato,
> I think you are talking about how we organize different operators into
> map-reduce jobs. Unfortunately there is no document currently. Basically we
> will put as much operators into one map-reduce job as possible.
> Co-group/Group, Join, Order, Distinct, Cross, Stream will create a
> map-reduce boundary; Most others we will put into existing jobs. The main
> logic is inside MRCompiler.java.
>
>
> Daniel
>
> Renato Marroquín Mogrovejo wrote:
>
>> Anyone, please?
>>
>> Renato M.
>>
>> 2010/8/24 Renato Marroquín Mogrovejo <renatoj.marroquin@gmail.com>
>>
>>
>>
>>> Hi Daniel,
>>>
>>> Thanks, but that was not what I was actually looking. What I want to know
>>> is for example, how the optimizer work when the bags' logical plans are
>>> combined, or if all commands are reduced at the end to CO-GROUP commands,
>>> how is this handled? I know from Pig's paper that the ORDER, and LOAD,
>>> commands generate new MapReduce jobs, are there any optimizations for the
>>> physical plans?
>>> Thanks in advanced.
>>>
>>>
>>> Renato M.
>>>
>>> 2010/8/23 Daniel Dai <jianyong@yahoo-inc.com>
>>>
>>> Hi, Renato,
>>>
>>>
>>>> There is a description of optimization rule in Pig Latin reference menu:
>>>>
>>>> http://hadoop.apache.org/pig/docs/r0.7.0/piglatin_ref1.html#Optimization+Rules
>>>> .
>>>> Is that enough?
>>>>
>>>> Daniel
>>>>
>>>>
>>>> Renato Marroquín Mogrovejo wrote:
>>>>
>>>>
>>>>
>>>>> Hey everyone, I was wondering if anybody has any references or
>>>>> suggestion
>>>>> on
>>>>> how to learn about Pig's optimizer besides the source code or Pig's
>>>>> paper.
>>>>> Thanks in advance.
>>>>>
>>>>>
>>>>> Renato M.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message