hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-16852) PTF: RANK() re-evaluates order predicates on the reducer
Date Thu, 08 Jun 2017 01:53:18 GMT
Gopal V created HIVE-16852:
------------------------------

             Summary: PTF: RANK() re-evaluates order predicates on the reducer
                 Key: HIVE-16852
                 URL: https://issues.apache.org/jira/browse/HIVE-16852
             Project: Hive
          Issue Type: Bug
          Components: Physical Optimizer
    Affects Versions: 2.1.1, 3.0.0
            Reporter: Gopal V


{code}
explain select ss_item_sk, rank() over(order by cast(ss_list_price as decimal(38,10))) as
r , ss_list_price from store_sales;

STAGE PLANS:
  Stage: Stage-1
    Tez
      DagId: root_20170608015140_7b0debb9-b14b-4150-b004-9743c6127392:3
      Edges:
        Reducer 2 <- Map 1 (SIMPLE_EDGE)
      DagName:
      Vertices:
        Map 1
            Map Operator Tree:
                TableScan
                  alias: store_sales
                  Statistics: Num rows: 28800426268 Data size: 450435120648 Basic stats: COMPLETE
Column stats: COMPLETE
                  Reduce Output Operator
                    key expressions: 0 (type: int), CAST( ss_list_price AS decimal(38,10))
(type: decimal(38,10))
                    sort order: ++
                    Map-reduce partition columns: 0 (type: int)
                    Statistics: Num rows: 28800426268 Data size: 450435120648 Basic stats:
COMPLETE Column stats: COMPLETE
                    value expressions: ss_item_sk (type: bigint), ss_list_price (type: double)
            Execution mode: vectorized, llap
            LLAP IO: all inputs
        Reducer 2 
            Execution mode: llap
            Reduce Operator Tree:
              Select Operator
                expressions: VALUE._col1 (type: bigint), VALUE._col11 (type: double)
                outputColumnNames: _col1, _col11
                Statistics: Num rows: 28800426268 Data size: 8399352770616 Basic stats: COMPLETE
Column stats: COMPLETE
                PTF Operator
                  Function definitions:
                      Input definition
                        input alias: ptf_0
                        output shape: _col1: bigint, _col11: double
                        type: WINDOWING
                      Windowing table definition
                        input alias: ptf_1
                        name: windowingtablefunction
                        order by: CAST( _col11 AS decimal(38,10)) ASC NULLS FIRST
                        partition by: 0
                        raw input shape:
                        window functions:
                            window function definition
                              alias: rank_window_0
                              arguments: CAST( _col11 AS decimal(38,10))
                              name: rank
                              window function: GenericUDAFRankEvaluator
                              window frame: PRECEDING(MAX)~FOLLOWING(MAX)
                              isPivotResult: true
                  Statistics: Num rows: 28800426268 Data size: 8399352770616 Basic stats:
COMPLETE Column stats: COMPLETE
                  Select Operator
                    expressions: _col1 (type: bigint), rank_window_0 (type: int), _col11 (type:
double)
                    outputColumnNames: _col0, _col1, _col2
                    Statistics: Num rows: 28800426268 Data size: 565636825720 Basic stats:
COMPLETE Column stats: COMPLETE
                    File Output Operator
                      compressed: false
                      Statistics: Num rows: 28800426268 Data size: 565636825720 Basic stats:
COMPLETE Column stats: COMPLETE
                      table:
                          input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                          output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                          serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
{code}

This forces the Decimal cast to be evaluated ~2x - once to produce the KEY expression and
once within the window function.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message