Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Fri, 20 May 2016 17:56:13 +0000 (UTC)
From: "Wei Zheng (JIRA)" <jira@apache.org>
To: dev@hive.apache.org
Message-ID: <JIRA.12971596.1463766944000.251338.1463766973278@Atlassian.JIRA>
In-Reply-To: <JIRA.12971596.1463766944000@Atlassian.JIRA>
References: <JIRA.12971596.1463766944000@Atlassian.JIRA> <JIRA.12971596.1463766944090@arcas>
Subject: [jira] [Created] (HIVE-13809) Hybrid Grace Hash Join memory usage
 estimation didn't take into account the bloom filter size
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 20 May 2016 17:56:16 -0000

Wei Zheng created HIVE-13809:
--------------------------------

             Summary: Hybrid Grace Hash Join memory usage estimation didn't take into account the bloom filter size
                 Key: HIVE-13809
                 URL: https://issues.apache.org/jira/browse/HIVE-13809
             Project: Hive
          Issue Type: Bug
          Components: Hive
    Affects Versions: 2.0.0, 2.1.0
            Reporter: Wei Zheng
            Assignee: Wei Zheng


Memory estimation is important during hash table loading, because we need to make the decision of whether to load the next hash partition in memory or spill it. If the assumption is there's enough memory but it turns out not the case, we will run into OOM problem.

Currently hybrid grace hash join memory usage estimation didn't take into account the bloom filter size. In large test cases (TB scale) the bloom filter grows as big as hundreds of MB, big enough to cause estimation error.

The solution is to count in the bloom filter size into memory estimation.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)