hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (PIG-1416) explain does not show the inner plans of the MapReduce plan
Date Fri, 14 May 2010 00:45:42 GMT

     [ https://issues.apache.org/jira/browse/PIG-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thejas M Nair resolved PIG-1416.
--------------------------------

    Resolution: Invalid

The secondary key optimizer is removing the POSort, that is why it does not appear in MR plan.
Closing as invalid. 


> explain does not show the inner plans of the MapReduce plan
> -----------------------------------------------------------
>
>                 Key: PIG-1416
>                 URL: https://issues.apache.org/jira/browse/PIG-1416
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Thejas M Nair
>             Fix For: 0.8.0
>
>
> The  inner plan in MR plan is very useful in understanding query plans where the inner
plans are not present in the Physical or Logical plans. For example, in case of "order by",
the sampling MR job is not part of the physical plan, and the reduce has a POSort which is
not shown in explain . 
> In following example, notice that POSort is not shown the MR plan .
> {code}
> grunt> l = load 'file.txt' as (a, b, c);
> grunt> g = group l by a;
> grunt> f = foreach g { s = order l by $0; generate s; }
> grunt> explain f;
> grunt> explain f;
> #-----------------------------------------------
> # Logical Plan:
> #-----------------------------------------------
> Store 1-137 Schema: {s: {a: bytearray,b: bytearray,c: bytearray}} Type: Unknown
> |
> |---ForEach 1-136 Schema: {s: {a: bytearray,b: bytearray,c: bytearray}} Type: bag
>     |   |
>     |   Project 1-132 Projections:  [*]  Overloaded: false FieldSchema: s: bag({a: bytearray,b:
bytearray,c: bytearray}) Type: bag
>     |   Input: SORT 1-133|
>     |   |---SORT 1-133 Schema: {a: bytearray,b: bytearray,c: bytearray} Type: bag
>     |       |   |
>     |       |   Project 1-134 Projections: [0] Overloaded: false FieldSchema: a: bytearray
Type: bytearray
>     |       |   Input: Project 1-135 Projections: [1] Overloaded: true
>     |       |
>     |       |---Project 1-135 Projections: [1] Overloaded: true FieldSchema: l: tuple({a:
bytearray,b: bytearray,c: bytearray}) Type: tuple
>     |           Input: CoGroup 1-126
>     |
>     |---CoGroup 1-126 Schema: {group: bytearray,l: {a: bytearray,b: bytearray,c: bytearray}}
Type: bag
>         |   |
>         |   Project 1-125 Projections: [0] Overloaded: false FieldSchema: a: bytearray
Type: bytearray
>         |   Input: Load 1-124
>         |
>         |---Load 1-124 Schema: {a: bytearray,b: bytearray,c: bytearray} Type: bag
> #-----------------------------------------------
> # Physical Plan:
> #-----------------------------------------------
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-148
> |
> |---New For Each(false)[bag] - 1-147
>     |   |
>     |   RelationToExpressionProject[bag][*] - 1-146
>     |   |
>     |   |---POSort[bag]() - 1-145
>     |       |   |
>     |       |   Project[bytearray][0] - 1-144
>     |       |
>     |       |---Project[tuple][1] - 1-143
>     |
>     |---Package[tuple]{bytearray} - 1-140
>         |
>         |---Global Rearrange[tuple] - 1-139
>             |
>             |---Local Rearrange[tuple]{bytearray}(false) - 1-141
>                 |   |
>                 |   Project[bytearray][0] - 1-142
>                 |
>                 |---Load(file:///Users/tejas/trunk_oby/file.txt:org.apache.pig.builtin.PigStorage)
- 1-138
> 2010-05-13 15:47:32,102 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
> 2010-05-13 15:47:32,102 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
> #--------------------------------------------------
> # Map Reduce Plan
> #--------------------------------------------------
> MapReduce node 1-149
> Map Plan
> Local Rearrange[tuple]{bytearray}(false) - 1-141
> |   |
> |   Project[bytearray][0] - 1-142
> |
> |---Load(file:///Users/tejas/trunk_oby/file.txt:org.apache.pig.builtin.PigStorage) -
1-138--------
> Reduce Plan
> Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-148
> |
> |---New For Each(false)[bag] - 1-147
>     |   |
>     |   RelationToExpressionProject[bag][*] - 1-146
>     |   |
>     |   |---Project[tuple][1] - 1-143
>     |
>     |---Package[tuple]{bytearray} - 1-140--------
> Global sort: false
> ----------------
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message