hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Jaiswal <djais...@hortonworks.com>
Subject Re: Review Request 55156: Min-max runtime filtering
Date Thu, 19 Jan 2017 00:59:18 GMT


> On Jan. 13, 2017, 12:08 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java, line 582
> > <https://reviews.apache.org/r/55156/diff/6/?file=1602202#file1602202line582>
> >
> >     This might be more appropriate on the FilterOperator, as I believe there are
some instances where the predicates may not be pushed to the TableScan, but still exist in
the FilterOperator. Or possibly done on both - DynamicPartitionPruningOptimization seems to
apply it to both.
> 
> Deepak Jaiswal wrote:
>     Agreed. I will update in the next patch

As of current patch, if the TS does not have the predicate pushed down, the semijoin optimization
is not done. Otherwise, TS and the FilterOperator share the filter expression.
The code in DynamicPartitionPruningOptimization is actually dead code as it never actually
finds any predicate to be removed, not atleast in any of its tests.


- Deepak


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55156/#review161398
-----------------------------------------------------------


On Jan. 18, 2017, 8:42 p.m., Deepak Jaiswal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55156/
> -----------------------------------------------------------
> 
> (Updated Jan. 18, 2017, 8:42 p.m.)
> 
> 
> Review request for hive, Gopal V, Gunther Hagleitner, Jason Dere, Prasanth_J, and Rajesh
Balamohan.
> 
> 
> Bugs: HIVE-15269
>     https://issues.apache.org/jira/browse/HIVE-15269
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-15269 min-max runtime filtering.
> The patch also contains the patch for HIVE-15270.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 96928db 
>   itests/src/test/resources/testconfiguration.properties be5a747 
>   orc/src/test/org/apache/orc/impl/TestRecordReaderImpl.java cdd62ac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 69ba4a2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 940f2dd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DynamicValueRegistry.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeColumnEvaluator.java 24c8281 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantDefaultEvaluator.java 89a75eb

>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java 4fe72a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeDynamicValueEvaluator.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java b8d6ab7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java 0d03d8f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorHead.java 42685fb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorRef.java 0a6b66a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeFieldEvaluator.java ff32626 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java 221abd9

>   ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java bd0d28c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 6f01da0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java d98cea9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java ac5331e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 9718c48 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java 440e0a1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheWrapper.java 9768efa 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java 9049ddd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java 008f8a4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectCache.java 0141230 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 955fa80 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java 06dca00 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java d80f201 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 0cb6c8a

>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSMBMapJoinOperator.java 80b0a14

>   ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 9d900e4 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
26fcc45 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java 9e9beb0

>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/RedundantDynamicPruningConditionsRemoval.java
d9ce017 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
aa1e509 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java e2363eb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 35f34da 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RuntimeValuesInfo.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java e8b003e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java cdb9e1b 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java 1ecbaad 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 13a0811 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicValueDesc.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java 18d5285

>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java PRE-CREATION

>   ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestConvertAstToSearchArg.java 93b50a6

>   ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java 8cbc26d 
>   ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out d3acbcd 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out PRE-CREATION

>   storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/LiteralDelegate.java PRE-CREATION

>   storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentFactory.java 8fda95c

>   storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java 10d8c51

>   storage-api/src/java/org/apache/hive/common/util/BloomFilter.java e60690d 
> 
> Diff: https://reviews.apache.org/r/55156/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message