hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pengcheng Xiong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6998) Select query can only support maximum 128 distinct expressions
Date Thu, 04 Dec 2014 23:38:12 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234794#comment-14234794
] 

Pengcheng Xiong commented on HIVE-6998:
---------------------------------------

run with 129 and also 200 distinct expressions, no problem

hive> select count(distinct c0),count(distinct c1),count(distinct c2),count(distinct c3),count(distinct
c4),count(distinct c5),count(distinct c6),count(distinct c7),count(distinct c8),count(distinct
c9),count(distinct c10),count(distinct c11),count(distinct c12),count(distinct c13),count(distinct
c14),count(distinct c15),count(distinct c16),count(distinct c17),count(distinct c18),count(distinct
c19),count(distinct c20),count(distinct c21),count(distinct c22),count(distinct c23),count(distinct
c24),count(distinct c25),count(distinct c26),count(distinct c27),count(distinct c28),count(distinct
c29),count(distinct c30),count(distinct c31),count(distinct c32),count(distinct c33),count(distinct
c34),count(distinct c35),count(distinct c36),count(distinct c37),count(distinct c38),count(distinct
c39),count(distinct c40),count(distinct c41),count(distinct c42),count(distinct c43),count(distinct
c44),count(distinct c45),count(distinct c46),count(distinct c47),count(distinct c48),count(distinct
c49),count(distinct c50),count(distinct c51),count(distinct c52),count(distinct c53),count(distinct
c54),count(distinct c55),count(distinct c56),count(distinct c57),count(distinct c58),count(distinct
c59),count(distinct c60),count(distinct c61),count(distinct c62),count(distinct c63),count(distinct
c64),count(distinct c65),count(distinct c66),count(distinct c67),count(distinct c68),count(distinct
c69),count(distinct c70),count(distinct c71),count(distinct c72),count(distinct c73),count(distinct
c74),count(distinct c75),count(distinct c76),count(distinct c77),count(distinct c78),count(distinct
c79),count(distinct c80),count(distinct c81),count(distinct c82),count(distinct c83),count(distinct
c84),count(distinct c85),count(distinct c86),count(distinct c87),count(distinct c88),count(distinct
c89),count(distinct c90),count(distinct c91),count(distinct c92),count(distinct c93),count(distinct
c94),count(distinct c95),count(distinct c96),count(distinct c97),count(distinct c98),count(distinct
c99),count(distinct c100),count(distinct c101),count(distinct c102),count(distinct c103),count(distinct
c104),count(distinct c105),count(distinct c106),count(distinct c107),count(distinct c108),count(distinct
c109),count(distinct c110),count(distinct c111),count(distinct c112),count(distinct c113),count(distinct
c114),count(distinct c115),count(distinct c116),count(distinct c117),count(distinct c118),count(distinct
c119),count(distinct c120),count(distinct c121),count(distinct c122),count(distinct c123),count(distinct
c124),count(distinct c125),count(distinct c126),count(distinct c127),count(distinct c128),count(distinct
c129),count(distinct c130),count(distinct c131),count(distinct c132),count(distinct c133),count(distinct
c134),count(distinct c135),count(distinct c136),count(distinct c137),count(distinct c138),count(distinct
c139),count(distinct c140),count(distinct c141),count(distinct c142),count(distinct c143),count(distinct
c144),count(distinct c145),count(distinct c146),count(distinct c147),count(distinct c148),count(distinct
c149),count(distinct c150),count(distinct c151),count(distinct c152),count(distinct c153),count(distinct
c154),count(distinct c155),count(distinct c156),count(distinct c157),count(distinct c158),count(distinct
c159),count(distinct c160),count(distinct c161),count(distinct c162),count(distinct c163),count(distinct
c164),count(distinct c165),count(distinct c166),count(distinct c167),count(distinct c168),count(distinct
c169),count(distinct c170),count(distinct c171),count(distinct c172),count(distinct c173),count(distinct
c174),count(distinct c175),count(distinct c176),count(distinct c177),count(distinct c178),count(distinct
c179),count(distinct c180),count(distinct c181),count(distinct c182),count(distinct c183),count(distinct
c184),count(distinct c185),count(distinct c186),count(distinct c187),count(distinct c188),count(distinct
c189),count(distinct c190),count(distinct c191),count(distinct c192),count(distinct c193),count(distinct
c194),count(distinct c195),count(distinct c196),count(distinct c197),count(distinct c198),count(distinct
c199)from tbl_200columns;

OK
0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
Time taken: 1.727 seconds, Fetched: 1 row(s)

[~ashutoshc], I would like to close this jira ticket.

> Select query can only support maximum 128 distinct expressions
> --------------------------------------------------------------
>
>                 Key: HIVE-6998
>                 URL: https://issues.apache.org/jira/browse/HIVE-6998
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor, Serializers/Deserializers
>    Affects Versions: 0.14.0
>            Reporter: Chaoyu Tang
>
> Select query can only support maximum 128 distinct expressions. Otherwise, you will be
thrown ArrayIndexOutOfBoundsException. For a query like:
> select count(distinct c1),  count(distinct c2),  count(distinct c3),  count(distinct
c4),  count(distinct c5),  count(distinct c6), ...., count(distinct c128),  count(distinct
c129) from tbl_129columns;
> you will get error like:
> {code}
> java.lang.Exception: java.lang.RuntimeException: Hive Runtime Error while closing operators
>         at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)
> Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators
>         at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:260)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
>         at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>         at java.lang.Thread.run(Thread.java:695)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException:
-128
>         at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1141)
>         at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:579)
>         at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
>         at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
>         at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:591)
>         at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:227)
>         ... 10 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ArrayIndexOutOfBoundsException: -128
>         at org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1099)
>         at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1138)
>         ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsException:
-128
>         at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:327)
>         at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
>         at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1064)
>         at org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1082)
>         ... 16 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -128
>         at java.util.ArrayList.get(ArrayList.java:324)
>         at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:838)
>         at org.apache.hadoop.hive.serde2.binarysortable.BinarySortableSerDe.serialize(BinarySortableSerDe.java:600)
>         at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.toHiveKey(ReduceSinkOperator.java:401)
>         at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:320)
>         ... 19 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message