hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vineet Garg <vg...@hortonworks.com>
Subject Re: Review Request 54517: HIVE-15192: Subquery support with Calcite
Date Wed, 14 Dec 2016 19:18:52 GMT


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/lib/SubQueryWalker.java, line 27
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582079#file1582079line27>
> >
> >     Better name: Expression walker?

Updated class name to ExpressionWalker


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/lib/SubQueryWalker.java, line 40
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582079#file1582079line40>
> >
> >     When is later? 
> >     SubQueryExprProcessor already handles TOK_SUBQUERY_OP so why do we need to bypass
subquery processing?

Later is when we generate plan for subquery. (Shouldn't have said later since we have already
processed and created logical plan for subquery at this point). SubQueryWalker bypasses subquery
since we are seperately generating logical plan for it. SubQueryExprProcessor then uses this
generated pland and creates appropriate ExprNodeSubQueryDesc. I'll update comments to make
it more clear.


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java,
line 82
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582083#file1582083line82>
> >
> >     Each HiveFilter will be visited by RelShuttle. Why do we need to traverse the
tree here?

RelShuttle is used by decorrelation logic but SubqueryRemoveRule uses getVariableSet function
which needs to traverse the filter to get set of all corr vars.


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java,
line 283
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line283>
> >
> >     can be called before Decorrelator?

This particular rule is defined in HiveRelDecorrelator and is called just before decorrelation.
Edit: BTW we plan to use Calcite's RelDecorrelator soon and will have all these rules in decorrelation
anyway. Not sure if it is worth updating it now.


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java,
line 419
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line419>
> >
> >     This can be removed since we alreafy have overloaded version of this with HiveAggregate.

We can not remove this since at this point we have a plan with mix of LogicalAggregate and
HiveAggregate. Decorrelator uses reflection to dynamically figure out the type and call appropriate
method.


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java,
lines 1053-1054
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line1053>
> >
> >     Can this be removed? If so, what above comment above?

Yes removed.


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java,
line 860
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line860>
> >
> >     Can be removed.

Can't remove (same as above)


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java,
line 1144
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line1144>
> >
> >     Can be removed.

Can not (same as above)


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java,
line 1355
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line1355>
> >
> >     can be removed.

Cannot (reason is same as above)


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java,
line 77
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582087#file1582087line77>
> >
> >     This assert can fail if there is no subquery, no?

this rule is called only if there is subquery


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java,
line 513
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582089#file1582089line513>
> >
> >     this should be RexCorrelVariable, not RexFieldAccess? No?

No it consists of RexCorrelVariable (corExpr here)


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeSubQueryDesc.java, line 29
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582095#file1582095line29>
> >
> >     I am not sure about ExprNodeSubQueryDesc class since it encapsulates both calcite
as well as Hive data structure. Till Now, ExprNodeDesc is exclusively Hive concept. 
> >     Is it necessary to mix these two?

I am not sure how else could I encapsulate all subquery related info which is required later
to convert to RexSubQuery.


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/semijoin5.q, line 1
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582102#file1582102line1>
> >
> >     Any reason for this?

No I added it to make plan more readable and forgot to remove it. I'll get rid of it.


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/constant_prop_3.q.out, line 161
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582110#file1582110line161>
> >
> >     Plan change because of semi join flag?

No this has a NOT IN subquery


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/constprog_partitioner.q.out, lines 84-86
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582111#file1582111line84>
> >
> >     Semijoin flag?

No this has an IN subquery


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/masking_3.q.out, lines 19-21
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582125#file1582125line19>
> >
> >     Need to set semijoin flag in q file.

As we looked at it the other day this test has a subquery mask, that's why you see this plan
change.


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java,
line 1931
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line1931>
> >
> >     This should be Aggregate and Project instead.

It doesn't work with Aggregate and Project because rule's onMatch expects LogicalAggregate
and on receiving HiveAggregate it throws an exception of incomptabile types. These rules will
need to be re-written to support Hive node. I think re-writing decorrelate to instead work
with Aggregate, Project etc should resolve this issue.


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java,
line 2036
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line2036>
> >
> >     instanceof Filter, instead.

same as above


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveReplicatedRelBuilder.java,
lines 141-156
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582082#file1582082line141>
> >
> >     These should be Hive Rel Factories, not default.

Tried replacing these with HiveRel factories but bunch of tests started failing while creating
Project. I'll look into this later.


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java,
lines 214-218
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line214>
> >
> >     Do we need to fire all these rules? Some of the rules are already used during
optimization of calcite tree. Shall we just call those rules before calling Decorrelator?
Will avoid calling rules twice.

AdjustProjectForCountAggregateRule is implemented in HiveRelDecorrelator and all of these
rules are called before decorrelation. After decorrelation the plan looks very different and
it might be helpful calling Filter rules again. I am not yet sure if we need all of these
rules before decorrelation. I replicated this code from Calcite. I'll look into this.


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java,
lines 233-234
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582086#file1582086line233>
> >
> >     Similarly these rules can be called after decorrelator on whole tree from Calcite
planner.

These rules are called after decorrelation. Let me see if these rules are already called from
Calcite planner, if so we can get rid of these rules


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java,
line 64
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582083#file1582083line64>
> >
> >     Wont this assert fail (correctly) for non-correlated variables?

You are right. Looking into this


> On Dec. 13, 2016, 1:51 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java, line 2082
> > <https://reviews.apache.org/r/54517/diff/3/?file=1582090#file1582090line2082>
> >
> >     should generate a new id for new QB.

How do I generate a new id? I copied this code from previous subquery code.


- Vineet


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/54517/#review158889
-----------------------------------------------------------


On Dec. 12, 2016, 6:17 p.m., Vineet Garg wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/54517/
> -----------------------------------------------------------
> 
> (Updated Dec. 12, 2016, 6:17 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-15192
>     https://issues.apache.org/jira/browse/HIVE-15192
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> This patch is 1st phase of getting rid of Hive's subquery transformation and de-corelation
logic and leverage Calcite's functionality to plan sub-queries.
> 
> Known issues with this patch
> * Few return path tests are failing and are disabled with this patch.
> * Semi-join optimization (HIVE-15227) is disabled currently as it doesn't work with this
patch.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9064e49 
>   itests/src/test/resources/testconfiguration.properties aa3d72d 
>   ql/src/java/org/apache/hadoop/hive/ql/lib/SubQueryWalker.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelShuttle.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelShuttleImpl.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveReplicatedRelBuilder.java
PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveFilter.java
0410c91 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveJoin.java
ba9483e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveProject.java
3e0a9a6 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
d494c9f 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
f8fb475 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java f1f3bf9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java 3458fb6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 79e55b2 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckCtx.java 02896ff 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java ace3eaf 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeSubQueryDesc.java PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_nested_subquery.q  
>   ql/src/test/queries/clientnegative/subquery_restrictions.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_shared_alias.q  
>   ql/src/test/queries/clientpositive/cbo_rp_auto_join1.q cbfb5d5 
>   ql/src/test/queries/clientpositive/join31.q c79105f 
>   ql/src/test/queries/clientpositive/multiMapJoin2.q cf5dbb0 
>   ql/src/test/queries/clientpositive/semijoin5.q 3e7c20a 
>   ql/src/test/queries/clientpositive/subquery_in.q c01ae70 
>   ql/src/test/queries/clientpositive/subquery_notin.q 3f4fb7f 
>   ql/src/test/queries/clientpositive/subquery_notin_having.q 05148df 
>   ql/src/test/results/clientnegative/subquery_in_groupby.q.out 809bb0a 
>   ql/src/test/results/clientnegative/subquery_in_select.q.out 3d74132 
>   ql/src/test/results/clientnegative/subquery_nested_subquery.q.out 140b093 
>   ql/src/test/results/clientnegative/subquery_restrictions.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/constant_prop_3.q.out 58f1065 
>   ql/src/test/results/clientpositive/constprog_partitioner.q.out d1016ad 
>   ql/src/test/results/clientpositive/llap/cbo_rp_subq_in.q.out f6bfad2 
>   ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out f993cf0 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 70ec02f 
>   ql/src/test/results/clientpositive/llap/lineage3.q.out 1a532da 
>   ql/src/test/results/clientpositive/llap/subquery_exists.q.out 1a006d8 
>   ql/src/test/results/clientpositive/llap/subquery_in.q.out 321f1cc 
>   ql/src/test/results/clientpositive/llap/subquery_nested_subquery.q.out PRE-CREATION

>   ql/src/test/results/clientpositive/llap/subquery_notin.q.out 3da1acb 
>   ql/src/test/results/clientpositive/llap/subquery_shared_alias.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/subquery_views.q.out 35e80ae 
>   ql/src/test/results/clientpositive/llap/vector_groupby_mapjoin.q.out ff658d7 
>   ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out a075662 
>   ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
76c8404 
>   ql/src/test/results/clientpositive/masking_3.q.out 1925dce 
>   ql/src/test/results/clientpositive/masking_4.q.out 7e923e8 
>   ql/src/test/results/clientpositive/perf/query45.q.out 7bc137c 
>   ql/src/test/results/clientpositive/perf/query70.q.out 611af74 
>   ql/src/test/results/clientpositive/semijoin4.q.out 3c065a9 
>   ql/src/test/results/clientpositive/semijoin5.q.out 63b477c 
>   ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 567c6d3 
>   ql/src/test/results/clientpositive/spark/subquery_exists.q.out b58fcbe 
>   ql/src/test/results/clientpositive/spark/subquery_in.q.out 21a48ec 
>   ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 012c3eb 
>   ql/src/test/results/clientpositive/subq_where_serialization.q.out f689651 
>   ql/src/test/results/clientpositive/subquery_exists.q.out 86f9089 
>   ql/src/test/results/clientpositive/subquery_exists_having.q.out 8861c82 
>   ql/src/test/results/clientpositive/subquery_in_having.q.out 854aa36 
>   ql/src/test/results/clientpositive/subquery_notexists.q.out ede7855 
>   ql/src/test/results/clientpositive/subquery_notexists_having.q.out 9349f2d 
>   ql/src/test/results/clientpositive/subquery_notin_having.q.out 804f411 
>   ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out c7e1f02 
>   ql/src/test/results/clientpositive/vector_groupby_mapjoin.q.out 3468657 
>   ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 160b088 
> 
> Diff: https://reviews.apache.org/r/54517/diff/
> 
> 
> Testing
> -------
> 
> * Added new tests.
> * Pre-commit testing on-going
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message