hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gopal V <go...@hortonworks.com>
Subject Re: Review Request 61374: HIVE-17220: Bloomfilter probing in semijoin reduction is thrashing L1 dcache
Date Wed, 02 Aug 2017 20:26:11 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61374/#review182030
-----------------------------------------------------------




metastore/src/java/org/apache/hadoop/hive/metastore/hbase/AggrStatsInvalidatorFilter.java
Line 107 (original), 107 (patched)
<https://reviews.apache.org/r/61374/#comment257840>

    .toArray(bitset)?



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java
Line 113 (original), 106 (patched)
<https://reviews.apache.org/r/61374/#comment257842>

    use maxEntries?



ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java
Line 102 (original), 99 (patched)
<https://reviews.apache.org/r/61374/#comment257843>

    NonSyncArrayInputStream?



storage-api/src/java/org/apache/hive/common/util/BloomFilter.java
Line 79 (original), 76 (patched)
<https://reviews.apache.org/r/61374/#comment257844>

    leave comment about the lack of a copy - the data structure no longer copies, so any modification
to original array affects the BloomFilter.



storage-api/src/java/org/apache/hive/common/util/BloomKFilter.java
Lines 215 (patched)
<https://reviews.apache.org/r/61374/#comment257845>

    More comments on the bitPos computation needed to explain this Bloom filter impl.



storage-api/src/java/org/apache/hive/common/util/BloomKFilter.java
Lines 221 (patched)
<https://reviews.apache.org/r/61374/#comment257846>

    This won't unroll easily - better to use 1 loop + Arrays fill()


- Gopal V


On Aug. 2, 2017, 9:57 a.m., Prasanth_J wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61374/
> -----------------------------------------------------------
> 
> (Updated Aug. 2, 2017, 9:57 a.m.)
> 
> 
> Review request for hive and Gopal V.
> 
> 
> Bugs: HIVE-17220
>     https://issues.apache.org/jira/browse/HIVE-17220
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-17220: Bloomfilter probing in semijoin reduction is thrashing L1 dcache
> 
> 
> Diffs
> -----
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/AggrStatsInvalidatorFilter.java
4ca4229acdd2d5e45f1365bbe9e4c8f91d585168 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java
188a87e923ff6dba4484d7029f64f46ad87dc36f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilter.java
4b3eca093c00ebdfc558bd1d6126cd8fb55533ec 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/VectorUDAFBloomFilterMerge.java
67a7c508a3d12a2236e3b8d1789cb95f131fdb6b 
>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFBloomFilter.java 2413ae6d2648ff32c15d621466afb71c84f28ed1

>   ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFInBloomFilter.java 3e6e069548d0a7aefacfe8dba37ec7825889b1cf

>   storage-api/src/java/org/apache/hive/common/util/BloomFilter.java e9f419dcbe23f5044e2fb936944380c4ba226464

>   storage-api/src/java/org/apache/hive/common/util/BloomKFilter.java PRE-CREATION 
>   storage-api/src/test/org/apache/hive/common/util/TestBloomFilter.java e4ee93a01433550bfd9c6401ffc61be4d54e645d

>   storage-api/src/test/org/apache/hive/common/util/TestBloomKFilter.java PRE-CREATION

> 
> 
> Diff: https://reviews.apache.org/r/61374/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Prasanth_J
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message