hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julian Hyde (JIRA)" <>
Subject [jira] [Commented] (HIVE-12923) CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver groupby_grouping_sets4.q failure
Date Wed, 22 Feb 2017 20:11:44 GMT


Julian Hyde commented on HIVE-12923:

I'm thinking of an alternative solution to CALCITE-1069. Currently, as you know, an Aggregate
with more than one grouping set returns more columns than one with only one grouping set.
We have been arguing about whether there should be 1 extra column (Hive's preference) or N
extra columns (Calcite's preference).

My new proposal is that there should be no extra columns. We make GROUPING into an aggregate
function, and if you want those extra columns you can add calls to GROUPING.

If the row type of Aggregate is same regardless of the number of grouping sets, it will simplify
a bunch of things. For example, it would be easier to write a rule that pushes down the Filter
"group_id = 2", because we wouldn't have to worry about disappearing columns, and whether
they are used.

[~hsubramaniyan], [~jcamachorodriguez], Would the new proposal be acceptable to Hive?

> CBO: Calcite Operator To Hive Operator (Calcite Return Path): TestCliDriver groupby_grouping_sets4.q
> ------------------------------------------------------------------------------------------------------------
>                 Key: HIVE-12923
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: CBO
>            Reporter: Hari Sankar Sivarama Subramaniyan
>            Assignee: Hari Sankar Sivarama Subramaniyan
>         Attachments: HIVE-12923.1.patch, HIVE-12923.2.patch
> {code}
> (SELECT a, b, count(*) from T1 where a < 3 group by a, b with cube) subq1
> join
> (SELECT a, b, count(*) from T1 where a < 3 group by a, b with cube) subq2
> on subq1.a = subq2.a
> {code}
> Stack trace:
> {code}
> java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory.pruneJoinOperator(
>         at org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory.access$400(
>         at org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerJoinProc.process(
>         at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(
>         at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(
>         at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(
>         at org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(
>         at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(
>         at org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(
>         at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(
>         at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
>         at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
>         at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
>         at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(
>         at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
>         at org.apache.hadoop.hive.ql.Driver.compile(
>         at org.apache.hadoop.hive.ql.Driver.compile(
>         at org.apache.hadoop.hive.ql.Driver.compileInternal(
>         at org.apache.hadoop.hive.ql.Driver.runInternal(
>         at
>         at
>         at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(
>         at org.apache.hadoop.hive.cli.CliDriver.processCmd(
>         at org.apache.hadoop.hive.cli.CliDriver.processLine(
>         at org.apache.hadoop.hive.cli.CliDriver.processLine(
>         at org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(
>         at org.apache.hadoop.hive.ql.QTestUtil.executeClient(
>         at org.apache.hadoop.hive.cli.TestCliDriver.runTest(
>         at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_grouping_sets4(
> {code}

This message was sent by Atlassian JIRA

View raw message