calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Remus Rusanu <rrus...@hortonworks.com>
Subject Re: Wrong planner used in materialized view test? Assert Relational expression rel#14869... belongs to a different planner than is currently being used.
Date Thu, 02 Mar 2017 19:57:35 GMT
Hello Calcite devs,

Any hint whether there is an easy way to copy/clone a RelNode from a planner to another?

Thanks,
~Remus


On 3/1/17, 10:58 AM, "Julian Hyde" <jhyde@apache.org> wrote:

    1536 is a big task. It won’t get into 1.12.
    
    Jesus, In fact, you find that you need to do some work in that area to get MVs working
in Hive. Maryann kept bumping into issues in this area when she was using MVs for index optimization
in phoenix.
    
    Remus, You should ask on the list how to copy RelNodes to a new planner. I’m sure Maryann
knows.
    
    Julian
    
    
    
    > On Mar 1, 2017, at 6:01 AM, Jesus Camacho Rodriguez <jcamachorodriguez@hortonworks.com>
wrote:
    > 
    > LatticeTest seems to be re-parsing the lattice definition. In Hive, we
    > actually cache the plans in memory to avoid having to parse the view
    > definition at optimization time, so this is not an option.
    > 
    > I have not found any method to change the cluster/planner of a given
    > expression, and copy methods seem to be using the cluster of the nodes
    > that are being copied.
    > 
    > I wonder whether it is worth to wait for CALCITE-1536 to go in, and
    > revisit this issue then. Julian, did you plan CALCITE-1536 to be part
    > of 1.12?
    > 
    > 
    > Thanks,
    > Jesús
    > 
    > 
    > 
    > On 2/28/17, 6:59 PM, "Julian Hyde" <jhyde@apache.org> wrote:
    > 
    >> Did you try one of the RelNode.copy variants?
    >> 
    >> Or maybe whatever LatticeTest does?
    >> 
    >> On Tue, Feb 28, 2017 at 10:53 AM, Jesus Camacho Rodriguez
    >> <jcamachorodriguez@hortonworks.com> wrote:
    >>> Hi Julian,
    >>> 
    >>> Thanks for the pointers. Yes, I was checking and indeed we fail because the
    >>> materialized views nodes are registered with another planner.
    >>> 
    >>> *I think we currently solve the problem by copying the RelNodes that
    >>> define the MV, keeping all of their state but moving them to a
    >>> different planner.*
    >>> 
    >>> That is what I was trying to do, but is there a specific method for that?
    >>> 
    >>> Thanks,
    >>> Jesús
    >>> 
    >>> 
    >>> 
    >>> 
    >>> 
    >>> 
    >>> On 2/28/17, 6:49 PM, "Julian Hyde" <jhyde@apache.org> wrote:
    >>> 
    >>>> A RelNode belongs to a RelOptCluster, which belongs to a
    >>>> RelOptPlanner. This is helpful in most cases. But in the case of
    >>>> materialized views, the views (represented by a tree of RelNodes) live
    >>>> longer than a query.
    >>>> 
    >>>> I think we currently solve the problem by copying the RelNodes that
    >>>> define the MV, keeping all of their state but moving them to a
    >>>> different planner.
    >>>> 
    >>>> In https://issues.apache.org/jira/browse/CALCITE-1536 we're revisiting
    >>>> the planner life cycle. After it, you probably won't be able to get
    >>>> from RelNode -> planner. If you want the current planner, and are
in a
    >>>> rule, you'll get it via the RelOptRuleCall.
    >>>> 
    >>>> 
    >>>> On Tue, Feb 28, 2017 at 7:39 AM, Julian Hyde <jhyde.apache@gmail.com>
wrote:
    >>>>> In future please post questions like this to the Calcite dev list.
    >>>>> 
    >>>>> Julian
    >>>>> 
    >>>>> On Feb 28, 2017, at 07:24, Jesus Camacho Rodriguez
    >>>>> <jcamachorodriguez@hortonworks.com> wrote:
    >>>>> 
    >>>>> I have seen this issue before. While it is obvious what is going
on (using
    >>>>> different planners), I need to check why we currently hit the assert
but we
    >>>>> were not hitting it before.
    >>>>> 
    >>>>> I am going to start working again on materialized views support today,
so
    >>>>> I'll start with that issue and keep you posted.
    >>>>> 
    >>>>> Thanks,
    >>>>> Jesús
    >>>>> 
    >>>>> 
    >>>>> 
    >>>>> El 28 feb 2017, a las 14:55, Remus Rusanu <rrusanu@hortonworks.com>
    >>>>> escribió:
    >>>>> 
    >>>>> Hi Jesus et. al,
    >>>>> 
    >>>>> 
    >>>>> 
    >>>>> With Calcite 1.12 I’m hitting this assert when running
    >>>>> materialized_view_create_rewrite:
    >>>>> 
    >>>>> select * from cmv_mat_view
    >>>>> 
    >>>>> 
    >>>>> 
    >>>>> Relational expression
    >>>>> rel#14869:HiveProject.HIVE.[](input=rel#14867:HiveFilter.HIVE.[](input=rel#14826:HiveTableScan.HIVE.[](table=[default.cmv_basetable],table:alias=cmv_basetable)[false],condition==($0,
    >>>>> 2)),a=CAST(2):INTEGER,b=$1,c=$2) belongs to a different planner than
is
    >>>>> currently being used.
    >>>>> 
    >>>>> 
    >>>>> 
    >>>>> 
    >>>>> 
    >>>>> "87f1a29b-be8d-4d8f-9831-1fff5a710e72 main"
    >>>>> 
    >>>>>      at
    >>>>> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1475)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.calcite.plan.volcano.VolcanoPlanner.registerMaterializations(VolcanoPlanner.java:368)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:592)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1464)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1264)
    >>>>> 
    >>>>>      at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:1028)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1072)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1088)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:367)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11036)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:289)
    >>>>> 
    >>>>>      at
    >>>>> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
    >>>>> 
    >>>>> 
    >>>>> 
    >>>>> 
    >>>>> 
    >>>>> Sure enough, in debugger, the HiveProject operator planner is a different
    >>>>> instance of VolcanoPlanner from ‘this’ at the assert point. Before
I dive
    >>>>> deeper into this, knowing that there were problems recently discussed
around
    >>>>> materialized views, is this obviously related to some known issue?
    >>>>> 
    >>>>> 
    >>>>> 
    >>>>> Thanks,
    >>>>> 
    >>>>> ~Remus
    >>>> 
    >> 
    
    
    

Mime
View raw message