hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Jaiswal <djais...@hortonworks.com>
Subject Re: Review Request 55156: Min-max runtime filtering
Date Thu, 19 Jan 2017 01:38:58 GMT


> On Jan. 19, 2017, 12:07 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java, line 656
> > <https://reviews.apache.org/r/55156/diff/7-8/?file=1605144#file1605144line656>
> >
> >     remove this

Thanks for pointing this out. Taking out my name from it.


> On Jan. 19, 2017, 12:07 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java, line 806
> > <https://reviews.apache.org/r/55156/diff/7-8/?file=1605148#file1605148line806>
> >
> >     Even if the stats are non-null, you need to check that the stats are non-negative
- they are set to -1 if there are no stats available for the table.

Will do that. Thanks.


> On Jan. 19, 2017, 12:07 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java, line 807
> > <https://reviews.apache.org/r/55156/diff/7-8/?file=1605148#file1605148line807>
> >
> >     In the case of no stats, I would think it would be preferable to just drop the
entire semijoin opt.

Done!


> On Jan. 19, 2017, 12:07 a.m., Jason Dere wrote:
> > ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q, line 33
> > <https://reviews.apache.org/r/55156/diff/7-8/?file=1605154#file1605154line33>
> >
> >     redundant setting? See line 28

The 1st run is without the optimization is turned on.


> On Jan. 19, 2017, 12:07 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java, line 355
> > <https://reviews.apache.org/r/55156/diff/8/?file=1608127#file1608127line355>
> >
> >     Why was this moved?

I sent an email regarding this long back. With this change, the buffers are created after
the Evaluator is initialized. BloomFilter needs the evaluator to be initialized as it needs
the expected number of rows for that.


> On Jan. 19, 2017, 12:07 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java,
line 831
> > <https://reviews.apache.org/r/55156/diff/8/?file=1608145#file1608145line831>
> >
> >     Try to avoid doing formatting changes if you are not actually changing the code.
> >     Changes to formatting should be done in a separate Jira.

I have noticed that as well. I will review it the next time I create a patch and remove such
changes.


> On Jan. 19, 2017, 12:07 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java,
line 897
> > <https://reviews.apache.org/r/55156/diff/8/?file=1608145#file1608145line897>
> >
> >     Try to avoid doing formatting changes if you are not actually changing the code.

Above


> On Jan. 19, 2017, 12:07 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java,
line 906
> > <https://reviews.apache.org/r/55156/diff/8/?file=1608145#file1608145line906>
> >
> >     Try to avoid doing formatting changes if you are not actually changing the code.

above


> On Jan. 19, 2017, 12:07 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java, line 665
> > <https://reviews.apache.org/r/55156/diff/8/?file=1608146#file1608146line665>
> >
> >     Remove this

Removed my name from the log line.


> On Jan. 19, 2017, 12:07 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java, line
53
> > <https://reviews.apache.org/r/55156/diff/8/?file=1608155#file1608155line53>
> >
> >     Remove this

Done


> On Jan. 19, 2017, 12:07 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java,
line 104
> > <https://reviews.apache.org/r/55156/diff/8/?file=1608157#file1608157line104>
> >
> >     Change the cast type to PrimitiveObjectInspector

done. Thanks for pointing this out.


> On Jan. 19, 2017, 12:07 a.m., Jason Dere wrote:
> > ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out, line 705
> > <https://reviews.apache.org/r/55156/diff/8/?file=1608161#file1608161line705>
> >
> >     Most of the time, you should not change formatting on existing code if all you
are doing is changing whitespace.
> >     If we need to do formatting changes that can be done in a separate Jira.

As mentioned above, will take care of this in next patch.


- Deepak


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55156/#review162197
-----------------------------------------------------------


On Jan. 18, 2017, 8:42 p.m., Deepak Jaiswal wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55156/
> -----------------------------------------------------------
> 
> (Updated Jan. 18, 2017, 8:42 p.m.)
> 
> 
> Review request for hive, Gopal V, Gunther Hagleitner, Jason Dere, Prasanth_J, and Rajesh
Balamohan.
> 
> 
> Bugs: HIVE-15269
>     https://issues.apache.org/jira/browse/HIVE-15269
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-15269 min-max runtime filtering.
> The patch also contains the patch for HIVE-15270.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 96928db 
>   itests/src/test/resources/testconfiguration.properties be5a747 
>   orc/src/test/org/apache/orc/impl/TestRecordReaderImpl.java cdd62ac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 69ba4a2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 940f2dd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DynamicValueRegistry.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeColumnEvaluator.java 24c8281 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantDefaultEvaluator.java 89a75eb

>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java 4fe72a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeDynamicValueEvaluator.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java b8d6ab7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java 0d03d8f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorHead.java 42685fb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorRef.java 0a6b66a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeFieldEvaluator.java ff32626 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java 221abd9

>   ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java bd0d28c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 6f01da0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java d98cea9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java ac5331e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 9718c48 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java 440e0a1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheWrapper.java 9768efa 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java 9049ddd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java 008f8a4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectCache.java 0141230 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 955fa80 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java 06dca00 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java d80f201 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 0cb6c8a

>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSMBMapJoinOperator.java 80b0a14

>   ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 9d900e4 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
26fcc45 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java 9e9beb0

>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/RedundantDynamicPruningConditionsRemoval.java
d9ce017 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
aa1e509 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java e2363eb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 35f34da 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RuntimeValuesInfo.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java e8b003e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java cdb9e1b 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java 1ecbaad 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 13a0811 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicValue.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicValueDesc.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEvaluator.java 18d5285

>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java PRE-CREATION

>   ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestConvertAstToSearchArg.java 93b50a6

>   ql/src/test/org/apache/hadoop/hive/ql/io/sarg/TestSearchArgumentImpl.java 8cbc26d 
>   ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out d3acbcd 
>   ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction.q.out PRE-CREATION

>   storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/LiteralDelegate.java PRE-CREATION

>   storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentFactory.java 8fda95c

>   storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java 10d8c51

>   storage-api/src/java/org/apache/hive/common/util/BloomFilter.java e60690d 
> 
> Diff: https://reviews.apache.org/r/55156/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Deepak Jaiswal
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message