hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11306) Add a bloom-1 filter for Hybrid MapJoin spills
Date Wed, 22 Jul 2015 19:09:05 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637461#comment-14637461
] 

Gopal V commented on HIVE-11306:
--------------------------------

Looks like this might be due to the {{&& joinResult != JoinUtil.JoinResult.SPILL)}}
in the MapJoinOperator::process().

{code}
            if (!noOuterJoin) {
              // For Hybrid Grace Hash Join, during the 1st round processing,
              // we only keep the LEFT side if the row is not spilled
              if (!conf.isHybridHashJoin() || hybridMapJoinLeftover
                  || (!hybridMapJoinLeftover && joinResult != JoinUtil.JoinResult.SPILL))
{
                joinNeeded = true;
                storage[pos] = dummyObjVectors[pos];
              }
            } else {
              storage[pos] = emptyList;
            }
{code}

> Add a bloom-1 filter for Hybrid MapJoin spills
> ----------------------------------------------
>
>                 Key: HIVE-11306
>                 URL: https://issues.apache.org/jira/browse/HIVE-11306
>             Project: Hive
>          Issue Type: Improvement
>          Components: Hive
>    Affects Versions: 1.3.0, 2.0.0
>            Reporter: Gopal V
>            Assignee: Gopal V
>         Attachments: HIVE-11306.1.patch
>
>
> HIVE-9277 implemented Spillable joins for Tez, which suffers from a corner-case performance
issue when joining wide small tables against a narrow big table (like a user info table join
events stream).
> The fact that the wide table is spilled causes extra IO, even though the nDV of the join
key might be in the thousands.
> A cheap bloom-1 filter would add a massive performance gain for such queries, massively
cutting down on the spill IO costs for the big-table spills.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message