calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remus Rusanu <rrus...@hortonworks.com>
Subject Re: Cluster mismatch between RelNodes of a query and a materialized view
Date Wed, 08 Mar 2017 18:38:56 GMT
Thanks Amogh,  I will consider this, but replacing the planner inside cached tree (or cluster)
will require adding locking around access to the ‘one’ cached MV plan, and in the concurrent
HiveServer2 it process may lead to more problems (my biggest fear being deadlocks).

Thanks,
~Remus

On 3/8/17, 9:24 AM, "Amogh Margoor" <amoghm@qubole.com> wrote:

    We had created MaterializedCluster by extending RelOptCluster for solving
    this issue :
    
    https://github.com/qubole/quark/blob/master/optimizer/src/main/java/org/apache/calcite/plan/QuarkMaterializeCluster.java
    New cluster was used in RelNodes of Materialised Views and it allowed
    changing it's RelOptPlanner to that of RelNode being optimised. Caveat is
    that it will not work if there are concurrent optimisations of RelNodes
    happening in different threads using same copy of MaterializedViews.
    
    Thanks,
    Amogh
    
    
    On Wed, Mar 8, 2017 at 10:35 PM, Julian Hyde <jhyde@apache.org> wrote:
    
    > The argument should be a RelOptCluster, not a RelOptPlanner. The link from
    > RelNode to planner is indirect currently (via cluster) and will be
    > non-existent after CALCITE-1536.
    >
    > I question whether we need a new method. Putting an abstract method on
    > RelNode is a huge burden because every RelNode sub-class needs to be fixed
    > when people upgrade. Even a non-abstract method imposes a conceptual
    > burden: more methods to understand.
    >
    > So, my approach would be to sub-class RelShuttle. It’s sufficient that it
    > only works for LogicalXxx nodes.
    >
    > No need to copy RexNode expressions. They are immutable.
    >
    > Julian
    >
    >
    > > On Mar 8, 2017, at 4:14 AM, Remus Rusanu <rrusanu@hortonworks.com>
    > wrote:
    > >
    > > I created CALCITE-1681 https://issues.apache.org/
    > jira/browse/CALCITE-1681 and I intent to work on it for finishing
    > HIVE-15708.
    > > My current thinking is to create a RelCopier based on RelShuttle and add
    > a new abstract RelNode.copyTo(RelOptPlanner) that each concrete Rel type
    > must override. The Rex part is already handled by the existing RexCopier.
    > >
    > > Thanks,
    > > ~Remus
    > >
    > > On 3/6/17, 12:30 PM, "Julian Hyde" <jhyde@apache.org> wrote:
    > >
    > >    Every RelNode belongs to a RelOptCluster, and basically there is one
    > RelOptCluster created each time a query is prepared. When working with
    > materialized views, the view’s query is represented as a tree of RelNodes,
    > that tree is used for optimizing more than one query. When planning a
    > particular query, the nodes of that query will have a different
    > RelOptCluster than the nodes of the materialized view(s) they are matched
    > against.
    > >
    > >    How do we deal with this? Do we copy the nodes into the query’s
    > cluster once we have found a match? If so, how? I couldn’t find a sub-class
    > of RelVisitor or RelShuttle that copies trees to a different RelOptCluster.
    > >
    > >    By the way, https://issues.apache.org/jira/browse/CALCITE-1536 <
    > https://issues.apache.org/jira/browse/CALCITE-1536> aims to improve the
    > RelNode life-cycle but I don’t think it will solve this problem.
    > >
    > >    Julian
    > >
    > >
    >
    >
    

Mime
View raw message