hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Jaiswal <djais...@hortonworks.com>
Subject Re: Review Request 55156: Min-max runtime filtering
Date Fri, 13 Jan 2017 22:15:16 GMT


> On Jan. 13, 2017, 12:08 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java, line 765
> > <https://reviews.apache.org/r/55156/diff/4/?file=1599372#file1599372line765>
> >
> >     Maybe explain why the semijoin optimization should be removed in this situation

The code has been taken out as a proper solution is implemented to detect cycles due to mapside
joins and semijoin reduction.


> On Jan. 13, 2017, 12:08 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java, line 129
> > <https://reviews.apache.org/r/55156/diff/5-6/?file=1601605#file1601605line129>
> >
> >     Can you remove this call from runDynamicPartitionPruning() since it is being
called here as well?

Will do in next patch


> On Jan. 13, 2017, 12:08 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java, line 574
> > <https://reviews.apache.org/r/55156/diff/6/?file=1602202#file1602202line574>
> >
> >     OptimizeTezProcContext.pruningOpsRemovedByPriorOpt seems to be used in DynamicPartitionPruning
as a way to track which DPP operators have already been removed. Do we need to have something
similar to this for the semijoin optimizations?

Not sure if it is needed yet but certainly something to lookout.


> On Jan. 13, 2017, 12:08 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java, line 582
> > <https://reviews.apache.org/r/55156/diff/6/?file=1602202#file1602202line582>
> >
> >     This might be more appropriate on the FilterOperator, as I believe there are
some instances where the predicates may not be pushed to the TableScan, but still exist in
the FilterOperator. Or possibly done on both - DynamicPartitionPruningOptimization seems to
apply it to both.

Agreed. I will update in the next patch


- Deepak


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55156/#review161398
-----------------------------------------------------------


On Jan. 13, 2017, 9:51 p.m., Deepak Jaiswal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55156/
> -----------------------------------------------------------
> 
> (Updated Jan. 13, 2017, 9:51 p.m.)
> 
> 
> Review request for hive, Gopal V, Gunther Hagleitner, Jason Dere, Prasanth_J, and Rajesh
Balamohan.
> 
> 
> Bugs: HIVE-15269
>     https://issues.apache.org/jira/browse/HIVE-15269
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-15269 min-max runtime filtering.
> The patch also contains the patch for HIVE-15270.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c88ca87 
>   itests/src/test/resources/testconfiguration.properties be5a747 
>   orc/src/test/org/apache/orc/impl/TestRecordReaderImpl.java cdd62ac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 69ba4a2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 940f2dd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DynamicValueRegistry.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeColumnEvaluator.java 24c8281 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantDefaultEvaluator.java 89a75eb

>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java 4fe72a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeDynamicValueEvaluator.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java b8d6ab7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java 0d03d8f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorHead.java 42685fb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorRef.java 0a6b66a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeFieldEvaluator.java ff32626 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java 221abd9

>   ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java bd0d28c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java d98cea9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java ac5331e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 9718c48 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java 440e0a1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheWrapper.java 9768efa 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java 9049ddd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java 008f8a4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectCache.java 0141230 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 955fa80 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java 06dca00 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java d80f201 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 0cb6c8a

>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSMBMapJoinOperator.java 80b0a14

>   ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 9d900e4 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
26fcc45 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java 9e9beb0

>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/RedundantDynamicPruningConditionsRemoval.java
d9ce017 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
aa1e509 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java e2363eb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 35f34da 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RuntimeValuesInfo.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java e8b003e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java cdb9e1b 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 13a0811 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicValueDesc.java PRE-CREATION

>   ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestConvertAstToSearchArg.java 93b50a6

>   ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java 8cbc26d 
>   ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out PRE-CREATION

>   storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/LiteralDelegate.java PRE-CREATION

>   storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentFactory.java 8fda95c

>   storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java 10d8c51

> 
> Diff: https://reviews.apache.org/r/55156/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message