hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lefty Leverenz <leftylever...@gmail.com>
Subject Re: Review Request 58777: HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold
Date Thu, 04 May 2017 07:35:18 GMT


> On April 30, 2017, 8:12 a.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 3155-3158 (patched)
> > <https://reviews.apache.org/r/58777/diff/4/?file=1702345#file1702345line3155>
> >
> >     "after every these many rows" -> "after every interval of this many rows"
> >     
> >     "exceed" -> "exceeds"
> >     
> >     Also, please add another \n to the description after "when running" (line 3157
in diff revision 4).
> 
> Prasanth_J wrote:
>     Thanks Lefty! Fixed in the new patch.

+1  Thanks for the fixes, Prasanth.


- Lefty


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58777/#review173439
-----------------------------------------------------------


On May 2, 2017, 8:22 a.m., Prasanth_J wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58777/
> -----------------------------------------------------------
> 
> (Updated May 2, 2017, 8:22 a.m.)
> 
> 
> Review request for hive, Gunther Hagleitner, Sergey Shelukhin, and Siddharth Seth.
> 
> 
> Bugs: HIVE-16546
>     https://issues.apache.org/jira/browse/HIVE-16546
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-16546: LLAP: Fail map join tasks if hash table memory exceeds threshold
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/common/MemoryEstimate.java PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3400560 
>   llap-server/src/java/org/apache/hadoop/hive/llap/IncrementalObjectSizeEstimator.java
ff6e7ce 
>   ql/src/gen/vectorization/UDAFTemplates/VectorUDAFAvg.txt 4393c3b 
>   ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMax.txt 7468c2f 
>   ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxDecimal.txt 57b7ea5 
>   ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxIntervalDayTime.txt 749e97e

>   ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxString.txt 9dfc147 
>   ql/src/gen/vectorization/UDAFTemplates/VectorUDAFMinMaxTimestamp.txt 32ecb34 
>   ql/src/gen/vectorization/UDAFTemplates/VectorUDAFSum.txt bd0f14d 
>   ql/src/gen/vectorization/UDAFTemplates/VectorUDAFVar.txt dc9d4b1 
>   ql/src/gen/vectorization/UDAFTemplates/VectorUDAFVarDecimal.txt 01062a9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionError.java
PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionException.java
dbe00b6 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionHandler.java
7fc3226 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 595d1bd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
04e24bd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java a3bccc6

>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HybridHashTableContainer.java
04e89e8 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
c86e5f5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 6d71fef

>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 7b13e90 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java 486d43a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorAggregationBufferBatch.java
630046d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java 5b4c7c3

>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorAggregateExpression.java
0866f63 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFAvgDecimal.java
74e25ae 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFAvgTimestamp.java
483d9dc 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java
2139eae 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilterMerge.java
d2446d5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFCount.java
494febc 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFCountMerge.java
dec88cb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFCountStar.java
337ba0a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFStdPopTimestamp.java
8cd3506 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFStdSampTimestamp.java
61d6977 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFSumDecimal.java
b10f66f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFVarPopTimestamp.java
2709b07 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFVarSampTimestamp.java
03dce1e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMap.java
6242daf 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashMultiSet.java
1a41961 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashSet.java
331867c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastBytesHashTable.java
b93e977 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java
b6db3bc 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTableLoader.java
49ecdd1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastKeyStore.java
be51693 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMap.java
6fe98f9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashMultiSet.java
9140aee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashSet.java
d3efb11 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastLongHashTable.java
8bfa07c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMap.java
add4788 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashMultiSet.java
faefdbb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastMultiKeyHashSet.java
5328910 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMap.java
f13034f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashMultiSet.java
53ad7b4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastStringHashSet.java
723c729 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastTableContainer.java
05f1cf1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastValueStore.java
f9c5b34 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/hashtable/VectorMapJoinHashTable.java
c7e585c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedHashSet.java
93a89d7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedHashTable.java
5fe7861 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/optimized/VectorMapJoinOptimizedStringHashSet.java
f921b9c 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java ad77e87 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java b2893e7 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/HiveOpConverter.java
d375d1b 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java
93b8a5d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenSparkSkewJoinProcessor.java
405c3ca 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 654f3b1 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 032c7bb 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 940630c 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java ec80641

>   ql/src/test/org/apache/hadoop/hive/ql/exec/mapjoin/TestMapJoinMemoryExhaustionHandler.java
16b5b17 
>   serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java a4ecd9f 
>   storage-api/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java 4a745e4 
> 
> 
> Diff: https://reviews.apache.org/r/58777/diff/6/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message