hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "liyunzhang_intel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-17383) ArrayIndexOutOfBoundsException in VectorGroupByOperator
Date Mon, 28 Aug 2017 06:48:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-17383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143439#comment-16143439
] 

liyunzhang_intel commented on HIVE-17383:
-----------------------------------------

[~lirui]: this passes in latest master(6be50b7) in my tez env. If there is some wrong with
the configuration, tell me!
query
{code}
set hive.cbo.enable=false;
set hive.user.install.directory=file:///tmp;
set fs.default.name=file:///;
set fs.defaultFS=file:///;
set tez.staging-dir=/tmp;
set tez.ignore.lib.uris=true;
set tez.runtime.optimize.local.fetch=true;
set tez.local.mode=true;
set hive.explain.user=false;
explain select count(*) from (select key from src group by key) s where s.key='98';
{code}
explain
{code}
STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
    Tez
      DagId: root_20170828023743_be3df7bf-49cc-4c71-a4a7-25814558804c:1
      Edges:
        Reducer 2 <- Map 1 (SIMPLE_EDGE)
        Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)
      DagName: root_20170828023743_be3df7bf-49cc-4c71-a4a7-25814558804c:1
      Vertices:
        Map 1 
            Map Operator Tree:
                TableScan
                  alias: src
                  Statistics: Num rows: 2906 Data size: 5812 Basic stats: COMPLETE Column
stats: NONE
                  Filter Operator
                    predicate: (key = '98') (type: boolean)
                    Statistics: Num rows: 1453 Data size: 2906 Basic stats: COMPLETE Column
stats: NONE
                    Select Operator
                      Statistics: Num rows: 1453 Data size: 2906 Basic stats: COMPLETE Column
stats: NONE
                      Group By Operator
                        keys: '98' (type: string)
                        mode: hash
                        outputColumnNames: _col0
                        Statistics: Num rows: 1453 Data size: 2906 Basic stats: COMPLETE Column
stats: NONE
                        Reduce Output Operator
                          key expressions: '98' (type: string)
                          sort order: +
                          Map-reduce partition columns: '98' (type: string)
                          Statistics: Num rows: 1453 Data size: 2906 Basic stats: COMPLETE
Column stats: NONE
        Reducer 2 
            Reduce Operator Tree:
              Group By Operator
                keys: '98' (type: string)
                mode: mergepartial
                outputColumnNames: _col0
                Statistics: Num rows: 726 Data size: 1452 Basic stats: COMPLETE Column stats:
NONE
                Select Operator
                  Statistics: Num rows: 726 Data size: 1452 Basic stats: COMPLETE Column stats:
NONE
                  Group By Operator
                    aggregations: count()
                    mode: hash
                    outputColumnNames: _col0
                    Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats:
NONE
                    Reduce Output Operator
                      sort order: 
                      Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats:
NONE
                      value expressions: _col0 (type: bigint)
        Reducer 3 
            Reduce Operator Tree:
              Group By Operator
                aggregations: count(VALUE._col0)
                mode: mergepartial
                outputColumnNames: _col0
                Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
                File Output Operator
                  compressed: false
                  Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats:
NONE
                  table:
                      input format: org.apache.hadoop.mapred.SequenceFileInputFormat
                      output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                      serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

  Stage: Stage-0
    Fetch Operator
      limit: -1
      Processor Tree:
        ListSink

{code}

> ArrayIndexOutOfBoundsException in VectorGroupByOperator
> -------------------------------------------------------
>
>                 Key: HIVE-17383
>                 URL: https://issues.apache.org/jira/browse/HIVE-17383
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Rui Li
>
> Query to reproduce:
> {noformat}
> set hive.cbo.enable=false;
> select count(*) from (select key from src group by key) s where s.key='98';
> {noformat}
> The stack trace is:
> {noformat}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
> 	at org.apache.hadoop.hive.ql.exec.vector.VectorGroupKeyHelper.copyGroupKey(VectorGroupKeyHelper.java:107)
> 	at org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeReduceMergePartial.doProcessBatch(VectorGroupByOperator.java:831)
> 	at org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:174)
> 	at org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1046)
> 	at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:462)
> 	... 18 more
> {noformat}
> More details can be found in HIVE-16823



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message