calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remus Rusanu <rrus...@hortonworks.com>
Subject Diff in Materialization views registration between Calcite 1.10 and calcite 1.12
Date Thu, 09 Mar 2017 21:09:21 GMT
Moving to calcite-dev

From: Remus Rusanu <rrusanu@hortonworks.com>
Date: Thursday, March 9, 2017 at 1:04 PM
To: Ashutosh Chauhan <ashutosh@hortonworks.com>, Julian Hyde <jhyde@hortonworks.com>
Cc: "sqlopt@hortonworks.com" <sqlopt@hortonworks.com>
Subject: Why Calcite 1.10 is not hitting the assert

The 1.12 relevant assert stack is this:
       at org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1475)
       at org.apache.calcite.plan.volcano.VolcanoPlanner.registerMaterializations(VolcanoPlanner.java:368)
       at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:592)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1467)

In 1.10 the names are a bit different, but VolcanoPlanner.findBestExp() calls useApplicableMaterializations()
which exists immediately, because context.unwrap(CalciteConnectionConfig.class) returns null.
So no ‘registration’ occurs (registerImpl is never called with the provided materialization
plan, as per my debugging).

However, when needed, the materialization is found. This stack bellow finds it, and uses it,
despite not being ‘registered’:
       at org.apache.calcite.plan.volcano.VolcanoPlanner.getMaterializations(VolcanoPlanner.java:348)
       at org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveMaterializedViewFilterScanRule.apply(HiveMaterializedViewFilterScanRule.java:71)
       at org.apache.hadoop.hive.ql.optimizer.calcite.rules.views.HiveMaterializedViewFilterScanRule.onMatch(HiveMaterializedViewFilterScanRule.java:64)
       at org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:213)
       at org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:819)
       at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1455)

The result is the desired one:

hive> create materialized view srcm enable rewrite as select key from src where key=10;
…
hive> explain extended select key from src where key=10;
OK
STAGE DEPENDENCIES:
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-0
    Fetch Operator
      limit: -1
      Processor Tree:
        TableScan
          alias: default.srcm
          GatherStats: false
          Select Operator
            expressions: key (type: string)
            outputColumnNames: _col0
            ListSink

The big changes came with CALCITE-1500

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message