hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6429) MapJoinKey has large memory overhead in typical cases
Date Thu, 20 Feb 2014 02:19:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906497#comment-13906497
] 

Sergey Shelukhin commented on HIVE-6429:
----------------------------------------

Measured memory savings; 1 million row hashtable; option one - double+int key, double+string
value.

Before (not sure why there's one less key, might be heap dump or profiler glitch)
|Class|Objects|Shallow Size|Retained Size|
|java.lang.Object[]|2999997|79999920|256383712|
|org.apache.hadoop.hive.serde2.io.DoubleWritable|2999997|71999928|71999928|
|byte[]|999999|32383968|32383968|
|java.util.HashMap$Entry|999999|31999968|352383616|
|java.util.ArrayList|999999|23999976|208383760|
|org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer|999999|23999976|232383736|
|org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer$NoCopyingArrayList|999999|23999976|160383808|
|org.apache.hadoop.io.LongWritable|999999|23999976|23999976|
|org.apache.hadoop.io.Text|999999|23999976|56383912|
|org.apache.hadoop.hive.ql.exec.persistence.MapJoinKey|999999|15999984|87999912|
|java.util.HashMap$Entry[]|1|8388624|360772272|
|java.util.HashMap|1|48|360772320|

After
|Class|Objects|Shallow Size|Retained Size|
|byte[]|2000000|72384000|72384000|
|java.lang.Object[]|2000000|56000000|184384000|
|org.apache.hadoop.hive.serde2.io.DoubleWritable|2000000|48000000|48000000|
|java.util.HashMap$Entry|1000000|32000000|320384000|
|java.util.ArrayList|1000000|24000000|208384000|
|org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer|1000000|24000000|232384000|
|org.apache.hadoop.hive.ql.exec.persistence.MapJoinRowContainer$NoCopyingArrayList|1000000|24000000|160384000|
|org.apache.hadoop.io.Text|1000000|24000000|56384000|
|org.apache.hadoop.hive.ql.exec.persistence.MapJoinKeyBytes|1000000|16000000|56000000|
|java.util.HashMap$Entry[]|1|8388624|328772016|
|java.util.HashMap|1|48|328772064|

Savings are ~9%.
Not as large as expected, as apparently Java has large overhead for byte arrays, 9-byte array
takes 32 bytes and 17-byte one has 40.
Still, reasonably good, perhaps the patch can be extended.
Or different hashtable used...

> MapJoinKey has large memory overhead in typical cases
> -----------------------------------------------------
>
>                 Key: HIVE-6429
>                 URL: https://issues.apache.org/jira/browse/HIVE-6429
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HIVE-6429.01.patch, HIVE-6429.WIP.patch, HIVE-6429.patch
>
>
> The only thing that MJK really needs it hashCode and equals (well, and construction),
so there's no need to have array of writables in there. Assuming all the keys for a table
have the same structure, for the common case where keys are primitive types, we can store
something like a byte array combination of keys to reduce the memory usage. Will probably
speed up compares too.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message