hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Charles Chen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-2382) Invalid predicate pushdown from incorrect column expression map for select operator generated by GROUP BY operation
Date Wed, 17 Aug 2011 19:05:27 GMT

     [ https://issues.apache.org/jira/browse/HIVE-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Charles Chen updated HIVE-2382:
-------------------------------

    Description: 
When a GROUP BY is specified, a select operator is added before the GROUP BY in SemanticAnalyzer.insertSelectAllPlanForGroupBy.
 Currently, the column expression map for this is set to the column expression map for the
parent operator.  This behavior is incorrect as, for example, the parent operator could rearrange
the order of the columns (_col0 => _col0, _col1 => _col2, _col2 => _col1) and the
new operator should not repeat this.

The predicate pushdown optimization uses the column expression map to track which columns
a filter expression refers to at different operators.  This results in a filter on incorrect
columns.

Here is a simple case of this going wrong: Using
{noformat}
create table invites (id int, foo int, bar int);
{noformat}
executing the query
{noformat}
explain select * from (select foo, bar from (select bar, foo from invites c union all select
bar, foo from invites d) b) a group by bar, foo having bar=1;
{noformat}
results in
{noformat}
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Alias -> Map Operator Tree:
        a-subquery1:b-subquery1:c 
          TableScan
            alias: c
            Filter Operator
              predicate:
                  expr: (foo = 1)
                  type: boolean
              Select Operator
                expressions:
                      expr: bar
                      type: int
                      expr: foo
                      type: int
                outputColumnNames: _col0, _col1
                Union
                  Select Operator
                    expressions:
                          expr: _col1
                          type: int
                          expr: _col0
                          type: int
                    outputColumnNames: _col0, _col1
                    Select Operator
                      expressions:
                            expr: _col0
                            type: int
                            expr: _col1
                            type: int
                      outputColumnNames: _col0, _col1
                      Group By Operator
                        bucketGroup: false
                        keys:
                              expr: _col1
                              type: int
                              expr: _col0
                              type: int
                        mode: hash
                        outputColumnNames: _col0, _col1
                        Reduce Output Operator
                          key expressions:
                                expr: _col0
                                type: int
                                expr: _col1
                                type: int
                          sort order: ++
                          Map-reduce partition columns:
                                expr: _col0
                                type: int
                                expr: _col1
                                type: int
                          tag: -1
        a-subquery2:b-subquery2:d 
          TableScan
            alias: d
            Filter Operator
              predicate:
                  expr: (foo = 1)
                  type: boolean
              Select Operator
                expressions:
                      expr: bar
                      type: int
                      expr: foo
                      type: int
                outputColumnNames: _col0, _col1
                Union
                  Select Operator
                    expressions:
                          expr: _col1
                          type: int
                          expr: _col0
                          type: int
                    outputColumnNames: _col0, _col1
                    Select Operator
                      expressions:
                            expr: _col0
                            type: int
                            expr: _col1
                            type: int
                      outputColumnNames: _col0, _col1
                      Group By Operator
                        bucketGroup: false
                        keys:
                              expr: _col1
                              type: int
                              expr: _col0
                              type: int
                        mode: hash
                        outputColumnNames: _col0, _col1
                        Reduce Output Operator
                          key expressions:
                                expr: _col0
                                type: int
                                expr: _col1
                                type: int
                          sort order: ++
                          Map-reduce partition columns:
                                expr: _col0
                                type: int
                                expr: _col1
                                type: int
                          tag: -1
      Reduce Operator Tree:
        Group By Operator
          bucketGroup: false
          keys:
                expr: KEY._col0
                type: int
                expr: KEY._col1
                type: int
          mode: mergepartial
          outputColumnNames: _col0, _col1
          Select Operator
            expressions:
                  expr: _col0
                  type: int
                  expr: _col1
                  type: int
            outputColumnNames: _col0, _col1
            File Output Operator
              compressed: false
              GlobalTableId: 0
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

  Stage: Stage-0
    Fetch Operator
      limit: -1
{noformat}
Note that the filter is now "foo = 1", while the correct behavior is to have "bar = 1".  If
we remove the group by, the behavior is correct.

  was:
When a GROUP BY is specified, a select operator is added before the GROUP BY in SemanticAnalyzer.insertSelectAllPlanForGroupBy.
 Currently, the column expression map for this is set to the column expression map for the
parent operator.  This behavior is incorrect as, for example, the parent operator could rearrange
the order of the columns (_col0 => _col0, _col1 => _col2, _col2 => _col1) and the
new operator should not repeat this.

The predicate pushdown optimization uses the column expression map to track which columns
a filter expression refers to at different operators.  This results in a filter on incorrect
columns.

Here is a simple case of this going wrong: Using
{noformat}
create table invites (id int, foo int, bar int)
{noformat}
executing the query
{noformat}
explain select * from (select foo, bar from (select bar, foo from invites c union all select
bar, foo from invites d) b) a group by bar, foo having bar=1;
{noformat}
results in
{noformat}
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-1
    Map Reduce
      Alias -> Map Operator Tree:
        a-subquery1:b-subquery1:c 
          TableScan
            alias: c
            Filter Operator
              predicate:
                  expr: (foo = 1)
                  type: boolean
              Select Operator
                expressions:
                      expr: bar
                      type: int
                      expr: foo
                      type: int
                outputColumnNames: _col0, _col1
                Union
                  Select Operator
                    expressions:
                          expr: _col1
                          type: int
                          expr: _col0
                          type: int
                    outputColumnNames: _col0, _col1
                    Select Operator
                      expressions:
                            expr: _col0
                            type: int
                            expr: _col1
                            type: int
                      outputColumnNames: _col0, _col1
                      Group By Operator
                        bucketGroup: false
                        keys:
                              expr: _col1
                              type: int
                              expr: _col0
                              type: int
                        mode: hash
                        outputColumnNames: _col0, _col1
                        Reduce Output Operator
                          key expressions:
                                expr: _col0
                                type: int
                                expr: _col1
                                type: int
                          sort order: ++
                          Map-reduce partition columns:
                                expr: _col0
                                type: int
                                expr: _col1
                                type: int
                          tag: -1
        a-subquery2:b-subquery2:d 
          TableScan
            alias: d
            Filter Operator
              predicate:
                  expr: (foo = 1)
                  type: boolean
              Select Operator
                expressions:
                      expr: bar
                      type: int
                      expr: foo
                      type: int
                outputColumnNames: _col0, _col1
                Union
                  Select Operator
                    expressions:
                          expr: _col1
                          type: int
                          expr: _col0
                          type: int
                    outputColumnNames: _col0, _col1
                    Select Operator
                      expressions:
                            expr: _col0
                            type: int
                            expr: _col1
                            type: int
                      outputColumnNames: _col0, _col1
                      Group By Operator
                        bucketGroup: false
                        keys:
                              expr: _col1
                              type: int
                              expr: _col0
                              type: int
                        mode: hash
                        outputColumnNames: _col0, _col1
                        Reduce Output Operator
                          key expressions:
                                expr: _col0
                                type: int
                                expr: _col1
                                type: int
                          sort order: ++
                          Map-reduce partition columns:
                                expr: _col0
                                type: int
                                expr: _col1
                                type: int
                          tag: -1
      Reduce Operator Tree:
        Group By Operator
          bucketGroup: false
          keys:
                expr: KEY._col0
                type: int
                expr: KEY._col1
                type: int
          mode: mergepartial
          outputColumnNames: _col0, _col1
          Select Operator
            expressions:
                  expr: _col0
                  type: int
                  expr: _col1
                  type: int
            outputColumnNames: _col0, _col1
            File Output Operator
              compressed: false
              GlobalTableId: 0
              table:
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

  Stage: Stage-0
    Fetch Operator
      limit: -1
{noformat}
Note that the filter is now "foo = 1", while the correct behavior is to have "bar = 1".


> Invalid predicate pushdown from incorrect column expression map for select operator generated
by GROUP BY operation
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-2382
>                 URL: https://issues.apache.org/jira/browse/HIVE-2382
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.6.0
>            Reporter: Charles Chen
>            Assignee: Charles Chen
>            Priority: Critical
>
> When a GROUP BY is specified, a select operator is added before the GROUP BY in SemanticAnalyzer.insertSelectAllPlanForGroupBy.
 Currently, the column expression map for this is set to the column expression map for the
parent operator.  This behavior is incorrect as, for example, the parent operator could rearrange
the order of the columns (_col0 => _col0, _col1 => _col2, _col2 => _col1) and the
new operator should not repeat this.
> The predicate pushdown optimization uses the column expression map to track which columns
a filter expression refers to at different operators.  This results in a filter on incorrect
columns.
> Here is a simple case of this going wrong: Using
> {noformat}
> create table invites (id int, foo int, bar int);
> {noformat}
> executing the query
> {noformat}
> explain select * from (select foo, bar from (select bar, foo from invites c union all
select bar, foo from invites d) b) a group by bar, foo having bar=1;
> {noformat}
> results in
> {noformat}
> STAGE DEPENDENCIES:
>   Stage-1 is a root stage
>   Stage-0 is a root stage
> STAGE PLANS:
>   Stage: Stage-1
>     Map Reduce
>       Alias -> Map Operator Tree:
>         a-subquery1:b-subquery1:c 
>           TableScan
>             alias: c
>             Filter Operator
>               predicate:
>                   expr: (foo = 1)
>                   type: boolean
>               Select Operator
>                 expressions:
>                       expr: bar
>                       type: int
>                       expr: foo
>                       type: int
>                 outputColumnNames: _col0, _col1
>                 Union
>                   Select Operator
>                     expressions:
>                           expr: _col1
>                           type: int
>                           expr: _col0
>                           type: int
>                     outputColumnNames: _col0, _col1
>                     Select Operator
>                       expressions:
>                             expr: _col0
>                             type: int
>                             expr: _col1
>                             type: int
>                       outputColumnNames: _col0, _col1
>                       Group By Operator
>                         bucketGroup: false
>                         keys:
>                               expr: _col1
>                               type: int
>                               expr: _col0
>                               type: int
>                         mode: hash
>                         outputColumnNames: _col0, _col1
>                         Reduce Output Operator
>                           key expressions:
>                                 expr: _col0
>                                 type: int
>                                 expr: _col1
>                                 type: int
>                           sort order: ++
>                           Map-reduce partition columns:
>                                 expr: _col0
>                                 type: int
>                                 expr: _col1
>                                 type: int
>                           tag: -1
>         a-subquery2:b-subquery2:d 
>           TableScan
>             alias: d
>             Filter Operator
>               predicate:
>                   expr: (foo = 1)
>                   type: boolean
>               Select Operator
>                 expressions:
>                       expr: bar
>                       type: int
>                       expr: foo
>                       type: int
>                 outputColumnNames: _col0, _col1
>                 Union
>                   Select Operator
>                     expressions:
>                           expr: _col1
>                           type: int
>                           expr: _col0
>                           type: int
>                     outputColumnNames: _col0, _col1
>                     Select Operator
>                       expressions:
>                             expr: _col0
>                             type: int
>                             expr: _col1
>                             type: int
>                       outputColumnNames: _col0, _col1
>                       Group By Operator
>                         bucketGroup: false
>                         keys:
>                               expr: _col1
>                               type: int
>                               expr: _col0
>                               type: int
>                         mode: hash
>                         outputColumnNames: _col0, _col1
>                         Reduce Output Operator
>                           key expressions:
>                                 expr: _col0
>                                 type: int
>                                 expr: _col1
>                                 type: int
>                           sort order: ++
>                           Map-reduce partition columns:
>                                 expr: _col0
>                                 type: int
>                                 expr: _col1
>                                 type: int
>                           tag: -1
>       Reduce Operator Tree:
>         Group By Operator
>           bucketGroup: false
>           keys:
>                 expr: KEY._col0
>                 type: int
>                 expr: KEY._col1
>                 type: int
>           mode: mergepartial
>           outputColumnNames: _col0, _col1
>           Select Operator
>             expressions:
>                   expr: _col0
>                   type: int
>                   expr: _col1
>                   type: int
>             outputColumnNames: _col0, _col1
>             File Output Operator
>               compressed: false
>               GlobalTableId: 0
>               table:
>                   input format: org.apache.hadoop.mapred.TextInputFormat
>                   output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
>   Stage: Stage-0
>     Fetch Operator
>       limit: -1
> {noformat}
> Note that the filter is now "foo = 1", while the correct behavior is to have "bar = 1".
 If we remove the group by, the behavior is correct.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message