hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lefty Leverenz (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-12492) MapJoin: 4 million unique integers seems to be a probe plateau
Date Wed, 08 Mar 2017 04:12:37 GMT

    [ https://issues.apache.org/jira/browse/HIVE-12492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15887496#comment-15887496
] 

Lefty Leverenz edited comment on HIVE-12492 at 3/8/17 4:12 AM:
---------------------------------------------------------------

Doc note:  This adds *hive.auto.convert.join.hashtable.max.entries* to HiveConf.java, so it
needs to be documented in the wiki.

* [Configuration Properties -- Query and DDL Execution | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution]

Added a TODOC2.2 label.

Edit (7/Mar/17):  HIVE-16137 changes the default value to 40,000,000 so that's what should
be documented in the wiki.

Typo alert:  In the parameter description, "does not take affect" should be "does not take
effect."  This can be corrected in the wiki.


was (Author: lefty@hortonworks.com):
Doc note:  This adds *hive.auto.convert.join.hashtable.max.entries* to HiveConf.java, so it
needs to be documented in the wiki.

* [Configuration Properties -- Query and DDL Execution | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-QueryandDDLExecution]

Added a TODOC2.2 label.

Typo alert:  In the parameter description, "does not take affect" should be "does not take
effect."  This can be corrected in the wiki.

> MapJoin: 4 million unique integers seems to be a probe plateau
> --------------------------------------------------------------
>
>                 Key: HIVE-12492
>                 URL: https://issues.apache.org/jira/browse/HIVE-12492
>             Project: Hive
>          Issue Type: Improvement
>          Components: Query Planning
>    Affects Versions: 1.3.0, 1.2.1, 2.0.0
>            Reporter: Gopal V
>            Assignee: Jesus Camacho Rodriguez
>              Labels: TODOC2.2
>             Fix For: 2.2.0
>
>         Attachments: HIVE-12492.01.patch, HIVE-12492.02.patch, HIVE-12492.patch
>
>
> After 4 million keys, the map-join implementation seems to suffer from a performance
degradation. 
> The hashtable build & probe time makes this very inefficient, even if the data is
very compact (i.e 2 ints).
> Falling back onto the shuffle join or bucket map-join is useful after 2^22 items.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message