hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jesus Camacho Rodriguez (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13809) Hybrid Grace Hash Join memory usage estimation didn't take into account the bloom filter size
Date Tue, 31 May 2016 19:04:13 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308377#comment-15308377
] 

Jesus Camacho Rodriguez commented on HIVE-13809:
------------------------------------------------

Removing 2.1.0 target as issue is not tagged as Critical/Blocker and the RC will be created
tomorrow. Please feel free to commit to branch-2.1 anyway and fix for 2.1.0 if this happens
before the release.

> Hybrid Grace Hash Join memory usage estimation didn't take into account the bloom filter
size
> ---------------------------------------------------------------------------------------------
>
>                 Key: HIVE-13809
>                 URL: https://issues.apache.org/jira/browse/HIVE-13809
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 2.0.0, 2.1.0
>            Reporter: Wei Zheng
>            Assignee: Wei Zheng
>
> Memory estimation is important during hash table loading, because we need to make the
decision of whether to load the next hash partition in memory or spill it. If the assumption
is there's enough memory but it turns out not the case, we will run into OOM problem.
> Currently hybrid grace hash join memory usage estimation didn't take into account the
bloom filter size. In large test cases (TB scale) the bloom filter grows as big as hundreds
of MB, big enough to cause estimation error.
> The solution is to count in the bloom filter size into memory estimation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message