hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] Created: (PIG-1416) explain does not show the inner plans of the MapReduce plan
Date Thu, 13 May 2010 23:05:56 GMT
explain does not show the inner plans of the MapReduce plan
-----------------------------------------------------------

                 Key: PIG-1416
                 URL: https://issues.apache.org/jira/browse/PIG-1416
             Project: Pig
          Issue Type: Bug
          Components: impl
            Reporter: Thejas M Nair
             Fix For: 0.8.0


The  inner plan in MR plan is very useful in understanding query plans where the inner plans
are not present in the Physical or Logical plans. For example, in case of "order by", the
sampling MR job is not part of the physical plan, and the reduce has a POSort which is not
shown in explain . 

In following example, notice that POSort is not shown the MR plan .

{code}

grunt> l = load 'file.txt' as (a, b, c);
grunt> g = group l by a;
grunt> f = foreach g { s = order l by $0; generate s; }
grunt> explain f;

grunt> explain f;
#-----------------------------------------------
# Logical Plan:
#-----------------------------------------------
Store 1-137 Schema: {s: {a: bytearray,b: bytearray,c: bytearray}} Type: Unknown
|
|---ForEach 1-136 Schema: {s: {a: bytearray,b: bytearray,c: bytearray}} Type: bag
    |   |
    |   Project 1-132 Projections:  [*]  Overloaded: false FieldSchema: s: bag({a: bytearray,b:
bytearray,c: bytearray}) Type: bag
    |   Input: SORT 1-133|
    |   |---SORT 1-133 Schema: {a: bytearray,b: bytearray,c: bytearray} Type: bag
    |       |   |
    |       |   Project 1-134 Projections: [0] Overloaded: false FieldSchema: a: bytearray
Type: bytearray
    |       |   Input: Project 1-135 Projections: [1] Overloaded: true
    |       |
    |       |---Project 1-135 Projections: [1] Overloaded: true FieldSchema: l: tuple({a:
bytearray,b: bytearray,c: bytearray}) Type: tuple
    |           Input: CoGroup 1-126
    |
    |---CoGroup 1-126 Schema: {group: bytearray,l: {a: bytearray,b: bytearray,c: bytearray}}
Type: bag
        |   |
        |   Project 1-125 Projections: [0] Overloaded: false FieldSchema: a: bytearray Type:
bytearray
        |   Input: Load 1-124
        |
        |---Load 1-124 Schema: {a: bytearray,b: bytearray,c: bytearray} Type: bag

#-----------------------------------------------
# Physical Plan:
#-----------------------------------------------
Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-148
|
|---New For Each(false)[bag] - 1-147
    |   |
    |   RelationToExpressionProject[bag][*] - 1-146
    |   |
    |   |---POSort[bag]() - 1-145
    |       |   |
    |       |   Project[bytearray][0] - 1-144
    |       |
    |       |---Project[tuple][1] - 1-143
    |
    |---Package[tuple]{bytearray} - 1-140
        |
        |---Global Rearrange[tuple] - 1-139
            |
            |---Local Rearrange[tuple]{bytearray}(false) - 1-141
                |   |
                |   Project[bytearray][0] - 1-142
                |
                |---Load(file:///Users/tejas/trunk_oby/file.txt:org.apache.pig.builtin.PigStorage)
- 1-138

2010-05-13 15:47:32,102 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2010-05-13 15:47:32,102 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
#--------------------------------------------------
# Map Reduce Plan
#--------------------------------------------------
MapReduce node 1-149
Map Plan
Local Rearrange[tuple]{bytearray}(false) - 1-141
|   |
|   Project[bytearray][0] - 1-142
|
|---Load(file:///Users/tejas/trunk_oby/file.txt:org.apache.pig.builtin.PigStorage) - 1-138--------
Reduce Plan
Store(fakefile:org.apache.pig.builtin.PigStorage) - 1-148
|
|---New For Each(false)[bag] - 1-147
    |   |
    |   RelationToExpressionProject[bag][*] - 1-146
    |   |
    |   |---Project[tuple][1] - 1-143
    |
    |---Package[tuple]{bytearray} - 1-140--------
Global sort: false
----------------

{code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message