hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vineet Garg <vg...@hortonworks.com>
Subject Re: Review Request 55553: HIVE-15544 Support for scalar subqueries in filter (WHERE/HAVING)
Date Thu, 19 Jan 2017 23:59:28 GMT


> On Jan. 19, 2017, 7:53 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java,
line 234
> > <https://reviews.apache.org/r/55553/diff/2/?file=1608674#file1608674line234>
> >
> >     Any reason for disabling this rule?

For nested scalar subqueries this rule was, for whatever reason, pushing filter on top of
LogicalCorrelate to within left input of LogicalCorrelate, which is wrong and causes an exception
during decorrelation. I'll add a comment and plan to open a JIRA to investigate why it is
doing so.


> On Jan. 19, 2017, 7:53 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/subquery_in.q, line 63
> > <https://reviews.apache.org/r/55553/diff/2/?file=1608698#file1608698line63>
> >
> >     Instead of removing you may add group by in subquery, unless this query pattern
is covered elsewhere in tests.

This was removed since now we disable corr IN/NOT IN queries with aggregate


> On Jan. 19, 2017, 7:53 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/subquery_in_having.q, line 43
> > <https://reviews.apache.org/r/55553/diff/2/?file=1608699#file1608699line43>
> >
> >     This satisifies all conditions: correlated and no implied gby. Will this still
wont work?

That criteria works for Scalar queries only for now. For IN/NOT IN we disable corr queries
with aggregate for now.


> On Jan. 19, 2017, 7:53 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/subquery_in_having.q, line 128
> > <https://reviews.apache.org/r/55553/diff/2/?file=1608699#file1608699line128>
> >
> >     Dont we allow correlated predicates here?

same reason as above.


> On Jan. 19, 2017, 7:53 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/subquery_notin.q, line 65
> > <https://reviews.apache.org/r/55553/diff/2/?file=1608700#file1608700line65>
> >
> >     add gby ?

this was remove because of same reason as above


- Vineet


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55553/#review162239
-----------------------------------------------------------


On Jan. 19, 2017, 11:58 p.m., Vineet Garg wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55553/
> -----------------------------------------------------------
> 
> (Updated Jan. 19, 2017, 11:58 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-15544
>     https://issues.apache.org/jira/browse/HIVE-15544
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> This patch provides support for scalar subqueries in filter clause. Note that this patch
disables the following:
> * IN/NOT IN correlated subqueries containing aggregates (HIVE checks for such queries
and throw an exception)
> * SCALAR correlated subqueries containing aggregates with non-equi join predicates on
correlated columns (HIVE throws an exception for such queries).
> 
> 
> Diffs
> -----
> 
>   itests/src/test/resources/testconfiguration.properties be5a747 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 6f01da0 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSubquerySemanticException.java
PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HivePlannerContext.java 8beb0dd

>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
a373cdd 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
f1e8ebd 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
8d2e535 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9f1b9d5 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g cd9adfc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java 24381b9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f275f6a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 6c30efd 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeSubQueryDesc.java aec331b 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSQCountCheck.java PRE-CREATION

>   ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/TestCBORuleFiredOnlyOnce.java
44e157b 
>   ql/src/test/queries/clientnegative/subquery_exists_implicit_gby.q 9013df6 
>   ql/src/test/queries/clientnegative/subquery_in_implicit_gby.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_notexists_implicit_gby.q 852b295 
>   ql/src/test/queries/clientnegative/subquery_scalar_implicit_gby.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_scalar_multi_columns.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_scalar_multi_rows.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_with_or_cond.q c2c3221 
>   ql/src/test/queries/clientpositive/perf/query1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query23.q e8ebd86 
>   ql/src/test/queries/clientpositive/perf/query30.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query6.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query81.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/subquery_exists.q cac8e1b 
>   ql/src/test/queries/clientpositive/subquery_in.q fe0c9c8 
>   ql/src/test/queries/clientpositive/subquery_in_having.q 40b7e32 
>   ql/src/test/queries/clientpositive/subquery_notin.q f9b5405 
>   ql/src/test/queries/clientpositive/subquery_scalar.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/subquery_unqualcolumnrefs.q bdfa648 
>   ql/src/test/results/clientnegative/subquery_exists_implicit_gby.q.out f7251e3 
>   ql/src/test/results/clientnegative/subquery_in_implicit_gby.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_multiple_cols_in_select.q.out 7a16bae 
>   ql/src/test/results/clientnegative/subquery_notexists_implicit_gby.q.out da38f5f 
>   ql/src/test/results/clientnegative/subquery_scalar_implicit_gby.q.out PRE-CREATION

>   ql/src/test/results/clientnegative/subquery_scalar_multi_columns.q.out PRE-CREATION

>   ql/src/test/results/clientnegative/subquery_scalar_multi_rows.q.out PRE-CREATION 
>   ql/src/test/results/clientnegative/subquery_windowing_corr.q.out dcd3026 
>   ql/src/test/results/clientnegative/subquery_with_or_cond.q.out d2d743d 
>   ql/src/test/results/clientpositive/llap/subquery_exists.q.out 35c64e7 
>   ql/src/test/results/clientpositive/llap/subquery_in.q.out 95c13e6 
>   ql/src/test/results/clientpositive/llap/subquery_multi.q.out 536bfdb 
>   ql/src/test/results/clientpositive/llap/subquery_notin.q.out 21dea74 
>   ql/src/test/results/clientpositive/llap/subquery_scalar.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query23.q.out 6d4cfca 
>   ql/src/test/results/clientpositive/perf/query30.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query6.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query81.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/subquery_exists.q.out 9c02d6a 
>   ql/src/test/results/clientpositive/subquery_in_having.q.out 125187a 
>   ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 1eb1b93 
> 
> Diff: https://reviews.apache.org/r/55553/diff/
> 
> 
> Testing
> -------
> 
> Added q tests
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message