hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liyunzhang_intel (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-16600) Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in multi_insert cases
Date Fri, 19 May 2017 08:54:05 GMT

     [ https://issues.apache.org/jira/browse/HIVE-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

liyunzhang_intel updated HIVE-16600:
------------------------------------
    Attachment: HIVE-16600.7.patch

[~lirui]: thanks for your review. update HIVE-16600.7.patch.
Changes:
add two test case,
1. order by limit + multi-insert
{code}
FROM (select key,value from src order by key limit 10) a
INSERT OVERWRITE TABLE e1
    SELECT key, value
INSERT OVERWRITE TABLE e2
    SELECT key;
{code}

we should disable parallel order by in this case, which matches expectation
{code}
 Stage-2 is a root stage [MAPRED]
  Stage-0 depends on stages: Stage-2 [MOVE]
  Stage-3 depends on stages: Stage-0 [STATS]
  Stage-1 depends on stages: Stage-2 [MOVE]
  Stage-4 depends on stages: Stage-1 [STATS]

STAGE PLANS:
  Stage: Stage-2
    Spark
      Edges:
        Reducer 2 <- Map 1 (SORT, 1)
      DagName: root_20170519161033_283bdbba-0f73-4db1-afba-4da456e3cc55:1
      Vertices:
        Map 1
            Map Operator Tree:
                TableScan
                  alias: src
                  Statistics: Num rows: 29 Data size: 5812 Basic stats: COMPLETE Column stats:
NONE
                  GatherStats: false
                  Select Operator
                    expressions: key (type: string), value (type: string)
                    outputColumnNames: _col0, _col1
                    Statistics: Num rows: 29 Data size: 5812 Basic stats: COMPLETE Column
stats: NONE
                    Reduce Output Operator
                      key expressions: _col0 (type: string)
                      null sort order: a
                      sort order: +
                      Statistics: Num rows: 29 Data size: 5812 Basic stats: COMPLETE Column
stats: NONE
                      tag: -1
                      TopN: 10
                      TopN Hash Memory Usage: 0.1
                      value expressions: _col1 (type: string)
                      auto parallelism: false
            Path -> Alias:
              hdfs://bdpe41:8020/user/hive/warehouse/src [src]
            Path -> Partition:
              hdfs://bdpe41:8020/user/hive/warehouse/src
                Partition
                  base file name: src
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  properties:
                    bucket_count -1
                    column.name.delimiter ,
                    columns key,value
                    columns.comments 'default','default'
                    columns.types string:string
                    file.inputformat org.apache.hadoop.mapred.TextInputFormat
                    file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                    location hdfs://bdpe41:8020/user/hive/warehouse/src
                    name default.src
                    numFiles 1
                    numRows 0
                    rawDataSize 0
                    serialization.ddl struct src { string key, string value}
                    serialization.format 1
                    serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                    totalSize 5812
                    transient_lastDdlTime 1493960133
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

                    input format: org.apache.hadoop.mapred.TextInputFormat
                    output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                    properties:
                      bucket_count -1
                      column.name.delimiter ,
                      columns key,value
                      columns.comments 'default','default'
                      columns.types string:string
                      file.inputformat org.apache.hadoop.mapred.TextInputFormat
                      file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                      location hdfs://bdpe41:8020/user/hive/warehouse/src
                      name default.src
                      numFiles 1
                      numRows 0
                      rawDataSize 0
                      serialization.ddl struct src { string key, string value}
                      serialization.format 1
                      serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                      totalSize 5812
                      transient_lastDdlTime 1493960133
                    serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                    name: default.src
                  name: default.src
            Truncated Path -> Alias:
              /src [src]
        Reducer 2
            Needs Tagging: false
            Reduce Operator Tree:
              Select Operator
                expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string)
                outputColumnNames: _col0, _col1
                Statistics: Num rows: 29 Data size: 5812 Basic stats: COMPLETE Column stats:
NONE
                Limit
                  Number of rows: 10
                  Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE Column stats:
NONE
                  File Output Operator
                    compressed: false
                    GlobalTableId: 1
                    directory: hdfs://bdpe41:8020/user/hive/warehouse/e1/.hive-staging_hive_2017-05-19_16-10-33_636_2374207122339693520-1/-ext-10000
                    NumFilesPerFileSink: 1
                    Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE Column
stats: NONE
                    Stats Publishing Key Prefix: hdfs://bdpe41:8020/user/hive/warehouse/e1/.hive-staging_hive_2017-05-19_16-10-33_636_2374207122339693520-1/-ext-10000/
                    table:
                        input format: org.apache.hadoop.mapred.TextInputFormat
                        output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                        properties:
                          COLUMN_STATS_ACCURATE {"BASIC_STATS":"true"}
                          bucket_count -1
                          column.name.delimiter ,
                          columns key,value
                          columns.comments
                          columns.types string:string
                          file.inputformat org.apache.hadoop.mapred.TextInputFormat
                          file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                          location hdfs://bdpe41:8020/user/hive/warehouse/e1
                          name default.e1
                          numFiles 0
                          numRows 0
                          rawDataSize 0
                          serialization.ddl struct e1 { string key, string value}
                          serialization.format 1
                          serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                          totalSize 0
                          transient_lastDdlTime 1495181433
                        serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                        name: default.e1
                    TotalFiles: 1
                    GatherStats: true
                    MultiFileSpray: false
                  Select Operator
                    expressions: _col0 (type: string)
                    outputColumnNames: _col0
                    Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE Column
stats: NONE
                    File Output Operator
                      compressed: false
                      GlobalTableId: 2
                      directory: hdfs://bdpe41:8020/user/hive/warehouse/e2/.hive-staging_hive_2017-05-19_16-10-33_636_2374207122339693520-1/-ext-10002
                      NumFilesPerFileSink: 1
                      Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE Column
stats: NONE
                      Stats Publishing Key Prefix: hdfs://bdpe41:8020/user/hive/warehouse/e2/.hive-staging_hive_2017-05-19_16-10-33_636_2374207122339693520-1/-ext-10002/
                      table:
                          input format: org.apache.hadoop.mapred.TextInputFormat
                          output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                          properties:
                            COLUMN_STATS_ACCURATE {"BASIC_STATS":"true"}
                            bucket_count -1
                            column.name.delimiter ,
                            columns key
                            columns.comments
                            columns.types string
                            file.inputformat org.apache.hadoop.mapred.TextInputFormat
                            file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                            location hdfs://bdpe41:8020/user/hive/warehouse/e2
                            name default.e2
                            numFiles 0
                            numRows 0
                            rawDataSize 0
                            serialization.ddl struct e2 { string key}
                            serialization.format 1
                            serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                            totalSize 0
                            transient_lastDdlTime 1495181433
                          serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                          name: default.e2
                      TotalFiles: 1
                      GatherStats: true
                      MultiFileSpray: false

  Stage: Stage-0
    Move Operator
      tables:
          replace: true
          source: hdfs://bdpe41:8020/user/hive/warehouse/e1/.hive-staging_hive_2017-05-19_16-10-33_636_2374207122339693520-1/-ext-10000
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              properties:
                COLUMN_STATS_ACCURATE {"BASIC_STATS":"true"}
                bucket_count -1
                column.name.delimiter ,
                columns key,value
                columns.comments
                columns.types string:string
                file.inputformat org.apache.hadoop.mapred.TextInputFormat
                file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                location hdfs://bdpe41:8020/user/hive/warehouse/e1
                name default.e1
                numFiles 0
                numRows 0
                rawDataSize 0
                serialization.ddl struct e1 { string key, string value}
                serialization.format 1
                serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                totalSize 0
                transient_lastDdlTime 1495181433
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: default.e1

  Stage: Stage-3
    Stats-Aggr Operator
      Stats Aggregation Key Prefix: hdfs://bdpe41:8020/user/hive/warehouse/e1/.hive-staging_hive_2017-05-19_16-10-33_636_2374207122339693520-1/-ext-10000/

  Stage: Stage-1
    Move Operator
      tables:
          replace: true
          source: hdfs://bdpe41:8020/user/hive/warehouse/e2/.hive-staging_hive_2017-05-19_16-10-33_636_2374207122339693520-1/-ext-10002
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              properties:
                COLUMN_STATS_ACCURATE {"BASIC_STATS":"true"}
                bucket_count -1
                column.name.delimiter ,
                columns key
                columns.comments
                columns.types string
                file.inputformat org.apache.hadoop.mapred.TextInputFormat
                file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                location hdfs://bdpe41:8020/user/hive/warehouse/e2
                name default.e2
                numFiles 0
                numRows 0
                rawDataSize 0
                serialization.ddl struct e2 { string key}
                serialization.format 1
                serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                totalSize 0
                transient_lastDdlTime 1495181433
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: default.e2

  Stage: Stage-4
    Stats-Aggr Operator
      Stats Aggregation Key Prefix: hdfs://bdpe41:8020/user/hive/warehouse/e2/.hive-staging_hive_2017-05-19_16-10-33_636_2374207122339693520-1/-ext-10002/
{code}
2. orderby + multi-insert( 2 limit in sub query)
We should enable parallel order by, which matches expectation.
{code}
FROM (select key,value from src order by key) a
INSERT OVERWRITE TABLE e1
    SELECT key, value limit 10
INSERT OVERWRITE TABLE e2
    SELECT key limit 5;
{code}

explain
{code}
  Stage-0 depends on stages: Stage-2 [MOVE]
  Stage-3 depends on stages: Stage-0 [STATS]
  Stage-1 depends on stages: Stage-2 [MOVE]
  Stage-4 depends on stages: Stage-1 [STATS]

STAGE PLANS:
  Stage: Stage-2
    Spark
      Edges:
        Reducer 3 <- Reducer 5 (GROUP, 1)
        Reducer 4 <- Reducer 6 (GROUP, 1)
        Reducer 5 <- Map 1 (SORT, 46)
        Reducer 6 <- Map 1 (SORT, 46)
      DagName: root_20170519162519_7384dca1-7dc7-4186-99ae-d021eda761a6:1
      Vertices:
        Map 1
            Map Operator Tree:
                TableScan
                  alias: src
                  Statistics: Num rows: 29 Data size: 5812 Basic stats: COMPLETE Column stats:
NONE
                  GatherStats: false
                  Select Operator
                    expressions: key (type: string), value (type: string)
                    outputColumnNames: _col0, _col1
                    Statistics: Num rows: 29 Data size: 5812 Basic stats: COMPLETE Column
stats: NONE
                    Reduce Output Operator
                      key expressions: _col0 (type: string)
                      null sort order: a
                      sort order: +
                      Statistics: Num rows: 29 Data size: 5812 Basic stats: COMPLETE Column
stats: NONE
                      tag: -1
                      value expressions: _col1 (type: string)
                      auto parallelism: false
            Path -> Alias:
              hdfs://bdpe41:8020/user/hive/warehouse/src [src]
            Path -> Partition:
              hdfs://bdpe41:8020/user/hive/warehouse/src
                Partition
                  base file name: src
                  input format: org.apache.hadoop.mapred.TextInputFormat
                  output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                  properties:
                    bucket_count -1
                    column.name.delimiter ,
                    columns key,value
                    columns.comments 'default','default'
                    columns.types string:string
                    file.inputformat org.apache.hadoop.mapred.TextInputFormat
                    file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                    location hdfs://bdpe41:8020/user/hive/warehouse/src
                    name default.src
                    numFiles 1
                    numRows 0
                    rawDataSize 0
                    serialization.ddl struct src { string key, string value}
                    serialization.format 1
                    serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                    totalSize 5812
                    transient_lastDdlTime 1493960133
                  serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

                    input format: org.apache.hadoop.mapred.TextInputFormat
                    output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                    properties:
                      bucket_count -1
                      column.name.delimiter ,
                      columns key,value
                      columns.comments 'default','default'
                      columns.types string:string
                      file.inputformat org.apache.hadoop.mapred.TextInputFormat
                      file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                      location hdfs://bdpe41:8020/user/hive/warehouse/src
                      name default.src
                      numFiles 1
                      numRows 0
                      rawDataSize 0
                      serialization.ddl struct src { string key, string value}
                      serialization.format 1
                      serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                      totalSize 5812
                      transient_lastDdlTime 1493960133
                    serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                    name: default.src
                  name: default.src
            Truncated Path -> Alias:
              /src [src]
        Reducer 3
            Needs Tagging: false
            Reduce Operator Tree:
              Select Operator
                expressions: VALUE._col0 (type: string), VALUE._col1 (type: string)
                outputColumnNames: _col0, _col1
                Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE Column stats:
NONE
                Limit
                  Number of rows: 10
                  Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE Column stats:
NONE
                  File Output Operator
                    compressed: false
                    GlobalTableId: 1
                    directory: hdfs://bdpe41:8020/user/hive/warehouse/e1/.hive-staging_hive_2017-05-19_16-25-19_345_5343780662632806715-1/-ext-10000
                    NumFilesPerFileSink: 1
                    Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE Column
stats: NONE
                    Stats Publishing Key Prefix: hdfs://bdpe41:8020/user/hive/warehouse/e1/.hive-staging_hive_2017-05-19_16-25-19_345_5343780662632806715-1/-ext-10000/
                    table:
                        input format: org.apache.hadoop.mapred.TextInputFormat
                        output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                        properties:
                          COLUMN_STATS_ACCURATE {"BASIC_STATS":"true"}
                          bucket_count -1
                          column.name.delimiter ,
                          columns key,value
                          columns.comments
                          columns.types string:string
                          file.inputformat org.apache.hadoop.mapred.TextInputFormat
                          file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                          location hdfs://bdpe41:8020/user/hive/warehouse/e1
                          name default.e1
                          numFiles 0
                          numRows 0
                          rawDataSize 0
                          serialization.ddl struct e1 { string key, string value}
                          serialization.format 1
                          serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                          totalSize 0
                          transient_lastDdlTime 1495182319
                        serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                        name: default.e1
                    TotalFiles: 1
                    GatherStats: true
                    MultiFileSpray: false
        Reducer 4
            Needs Tagging: false
            Reduce Operator Tree:
              Select Operator
                expressions: VALUE._col0 (type: string)
                outputColumnNames: _col0
                Statistics: Num rows: 5 Data size: 1000 Basic stats: COMPLETE Column stats:
NONE
                Limit
                  Number of rows: 5
                  Statistics: Num rows: 5 Data size: 1000 Basic stats: COMPLETE Column stats:
NONE
                  File Output Operator
                    compressed: false
                    GlobalTableId: 2
                    directory: hdfs://bdpe41:8020/user/hive/warehouse/e2/.hive-staging_hive_2017-05-19_16-25-19_345_5343780662632806715-1/-ext-10002
                    NumFilesPerFileSink: 1
                    Statistics: Num rows: 5 Data size: 1000 Basic stats: COMPLETE Column stats:
NONE
                    Stats Publishing Key Prefix: hdfs://bdpe41:8020/user/hive/warehouse/e2/.hive-staging_hive_2017-05-19_16-25-19_345_5343780662632806715-1/-ext-10002/
                    table:
                        input format: org.apache.hadoop.mapred.TextInputFormat
                        output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                        properties:
                          COLUMN_STATS_ACCURATE {"BASIC_STATS":"true"}
                          bucket_count -1
                          column.name.delimiter ,
                          columns key
                          columns.comments
                          columns.types string
                          file.inputformat org.apache.hadoop.mapred.TextInputFormat
                          file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                          location hdfs://bdpe41:8020/user/hive/warehouse/e2
                          name default.e2
                          numFiles 0
                          numRows 0
                          rawDataSize 0
                          serialization.ddl struct e2 { string key}
                          serialization.format 1
                          serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                          totalSize 0
                          transient_lastDdlTime 1495182319
                        serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                        name: default.e2
                    TotalFiles: 1
                    GatherStats: true
                    MultiFileSpray: false
        Reducer 5
            Needs Tagging: false
            Reduce Operator Tree:
              Select Operator
                expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string)
                outputColumnNames: _col0, _col1
                Statistics: Num rows: 29 Data size: 5812 Basic stats: COMPLETE Column stats:
NONE
                Limit
                  Number of rows: 10
                  Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE Column stats:
NONE
                  Reduce Output Operator
                    null sort order:
                    sort order:
                    Statistics: Num rows: 10 Data size: 2000 Basic stats: COMPLETE Column
stats: NONE
                    tag: -1
                    TopN: 10
                    TopN Hash Memory Usage: 0.1
                    value expressions: _col0 (type: string), _col1 (type: string)
                    auto parallelism: false
        Reducer 6
            Needs Tagging: false
            Reduce Operator Tree:
              Select Operator
                expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string)
                outputColumnNames: _col0, _col1
                Statistics: Num rows: 29 Data size: 5812 Basic stats: COMPLETE Column stats:
NONE
                Select Operator
                  expressions: _col0 (type: string)
                  outputColumnNames: _col0
                  Statistics: Num rows: 29 Data size: 5812 Basic stats: COMPLETE Column stats:
NONE
                  Limit
                    Number of rows: 5
                    Statistics: Num rows: 5 Data size: 1000 Basic stats: COMPLETE Column stats:
NONE
                    Reduce Output Operator
                      null sort order:
                      sort order:
                      Statistics: Num rows: 5 Data size: 1000 Basic stats: COMPLETE Column
stats: NONE
                      tag: -1
                      TopN: 5
                      TopN Hash Memory Usage: 0.1
                      value expressions: _col0 (type: string)
                      auto parallelism: false

  Stage: Stage-0
    Move Operator
      tables:
          replace: true
          source: hdfs://bdpe41:8020/user/hive/warehouse/e1/.hive-staging_hive_2017-05-19_16-25-19_345_5343780662632806715-1/-ext-10000
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              properties:
                COLUMN_STATS_ACCURATE {"BASIC_STATS":"true"}
                bucket_count -1
                column.name.delimiter ,
                columns key,value
                columns.comments
                columns.types string:string
                file.inputformat org.apache.hadoop.mapred.TextInputFormat
                file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                location hdfs://bdpe41:8020/user/hive/warehouse/e1
                name default.e1
                numFiles 0
                numRows 0
                rawDataSize 0
                serialization.ddl struct e1 { string key, string value}
                serialization.format 1
                serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                totalSize 0
                transient_lastDdlTime 1495182319
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: default.e1

  Stage: Stage-3
    Stats-Aggr Operator
      Stats Aggregation Key Prefix: hdfs://bdpe41:8020/user/hive/warehouse/e1/.hive-staging_hive_2017-05-19_16-25-19_345_5343780662632806715-1/-ext-10000/

  Stage: Stage-1
    Move Operator
      tables:
          replace: true
          source: hdfs://bdpe41:8020/user/hive/warehouse/e2/.hive-staging_hive_2017-05-19_16-25-19_345_5343780662632806715-1/-ext-10002
          table:
              input format: org.apache.hadoop.mapred.TextInputFormat
              output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
              properties:
                COLUMN_STATS_ACCURATE {"BASIC_STATS":"true"}
                bucket_count -1
                column.name.delimiter ,
                columns key
                columns.comments
                columns.types string
                file.inputformat org.apache.hadoop.mapred.TextInputFormat
                file.outputformat org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
                location hdfs://bdpe41:8020/user/hive/warehouse/e2
                name default.e2
                numFiles 0
                numRows 0
                rawDataSize 0
                serialization.ddl struct e2 { string key}
                serialization.format 1
                serialization.lib org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
                totalSize 0
                transient_lastDdlTime 1495182319
              serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
              name: default.e2

  Stage: Stage-4
    Stats-Aggr Operator
      Stats Aggregation Key Prefix: hdfs://bdpe41:8020/user/hive/warehouse/e2/.hive-staging_hive_2017-05-19_16-25-19_345_5343780662632806715-1/-ext-10002/

{code}


> Refactor SetSparkReducerParallelism#needSetParallelism to enable parallel order by in
multi_insert cases
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-16600
>                 URL: https://issues.apache.org/jira/browse/HIVE-16600
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: liyunzhang_intel
>            Assignee: liyunzhang_intel
>         Attachments: HIVE-16600.1.patch, HIVE-16600.2.patch, HIVE-16600.3.patch, HIVE-16600.4.patch,
HIVE-16600.5.patch, HIVE-16600.6.patch, HIVE-16600.7.patch, mr.explain, mr.explain.log.HIVE-16600
>
>
> multi_insert_gby.case.q
> {code}
> set hive.exec.reducers.bytes.per.reducer=256;
> set hive.optimize.sampling.orderby=true;
> drop table if exists e1;
> drop table if exists e2;
> create table e1 (key string, value string);
> create table e2 (key string);
> FROM (select key, cast(key as double) as keyD, value from src order by key) a
> INSERT OVERWRITE TABLE e1
>     SELECT key, value
> INSERT OVERWRITE TABLE e2
>     SELECT key;
> select * from e1;
> select * from e2;
> {code} 
> the parallelism of Sort is 1 even we enable parallel order by("hive.optimize.sampling.orderby"
is set as "true").  This is not reasonable because the parallelism  should be calcuated by
 [Utilities.estimateReducers|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L170]
> this is because SetSparkReducerParallelism#needSetParallelism returns false when [children
size of RS|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java#L207]
is greater than 1.
> in this case, the children size of {{RS[2]}} is two.
> the logical plan of the case
> {code}
>    TS[0]-SEL[1]-RS[2]-SEL[3]-SEL[4]-FS[5]
>                             -SEL[6]-FS[7]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message