hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin" <ser...@hortonworks.com>
Subject Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory overhead
Date Thu, 13 Mar 2014 23:13:27 GMT


> On March 13, 2014, 6 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java,
line 31
> > <https://reviews.apache.org/r/18936/diff/2/?file=517002#file517002line31>
> >
> >     The ref longs are not just indexes, they contain bits from various components.

index in the array is the hash code


> On March 13, 2014, 6 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java,
line 173
> > <https://reviews.apache.org/r/18936/diff/2/?file=517002#file517002line173>
> >
> >     reserve -ve bytes?

it was supported, made cleaner


> On March 13, 2014, 6 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java,
line 477
> > <https://reviews.apache.org/r/18936/diff/2/?file=517002#file517002line477>
> >
> >     Really? :)

yes :) removed comment


- Sergey


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review36939
-----------------------------------------------------------


On March 12, 2014, 2:30 a.m., Sergey Shelukhin wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
> 
> (Updated March 12, 2014, 2:30 a.m.)
> 
> 
> Review request for hive, Gopal V and Gunther Hagleitner.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> See JIRA
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38 
>   ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3daf7a5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java
8854b19 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 61545b5

>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java
PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java a00aab3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java
PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db

>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java a8cb1ae

>   ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java
55b7415 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 84739ee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 6ecbcf7

>   ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java
65e3779 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java
755d783 
>   ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6 
>   ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984 
>   ql/src/test/results/clientpositive/tez/ctas.q.out aaac91c 
>   ql/src/test/results/clientpositive/tez/groupby3.q.out cb8f0e7 
>   ql/src/test/results/clientpositive/tez/insert1.q.out 7b8819d 
>   ql/src/test/results/clientpositive/tez/limit_pushdown.q.out 7d6b1c7 
>   ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out d420caf 
>   ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out bc2c650 
>   ql/src/test/results/clientpositive/tez/subquery_in.q.out c66602b 
>   serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29 
>   serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION 
>   serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java
bab505e 
>   serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd 
>   serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4 
>   serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6 
>   serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f

>   serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081

> 
> Diff: https://reviews.apache.org/r/18936/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Sergey Shelukhin
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message