hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zoltan Haindrich (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-19823) BytesBytesMultiHashMap estimation should account for loadFactor
Date Fri, 08 Jun 2018 12:55:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-19823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zoltan Haindrich updated HIVE-19823:
------------------------------------
    Description: 
it could happen that the capacity is known beforehand; and the estimated size of the hashtable
is accurate. but still; because after some time the element count violates loadfactor ratio
a rehash will occur.

this by default could happen with a {{1-loadfactor = 25%}}  probability
this rehashing takes aound 2 seconds on my system for 6.5M entries

https://github.com/apache/hive/blob/cfd57348c1ac188e0ba131d5636a62ff7b7c27be/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java#L176-L187

  was:
it could happen that the capacity is known beforehand; and the estimated size of the hashtable
is accurate; but still because after some time the element count violates loadfactor ration
a rehash will occur.

this by default could happen with a {{1-loadfactor = 25%}}  probability

https://github.com/apache/hive/blob/cfd57348c1ac188e0ba131d5636a62ff7b7c27be/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java#L176-L187


> BytesBytesMultiHashMap estimation should account for loadFactor
> ---------------------------------------------------------------
>
>                 Key: HIVE-19823
>                 URL: https://issues.apache.org/jira/browse/HIVE-19823
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Zoltan Haindrich
>            Assignee: Zoltan Haindrich
>            Priority: Major
>         Attachments: HIVE-19823.01.patch
>
>
> it could happen that the capacity is known beforehand; and the estimated size of the
hashtable is accurate. but still; because after some time the element count violates loadfactor
ratio a rehash will occur.
> this by default could happen with a {{1-loadfactor = 25%}}  probability
> this rehashing takes aound 2 seconds on my system for 6.5M entries
> https://github.com/apache/hive/blob/cfd57348c1ac188e0ba131d5636a62ff7b7c27be/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java#L176-L187



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message