Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Fri, 17 Jun 2016 10:53:05 +0000 (UTC)
From: "Hive QA (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.12971596.1463766944000.20154.1466160785189@Atlassian.JIRA>
In-Reply-To: <JIRA.12971596.1463766944000@Atlassian.JIRA>
References: <JIRA.12971596.1463766944000@Atlassian.JIRA> <JIRA.12971596.1463766944090@arcas>
Subject: [jira] [Commented] (HIVE-13809) Hybrid Grace Hash Join memory usage
 estimation didn't take into account the bloom filter size
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 17 Jun 2016 10:53:07 -0000


    [ https://issues.apache.org/jira/browse/HIVE-13809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15335857#comment-15335857 ] 

Hive QA commented on HIVE-13809:
--------------------------------


Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12811158/HIVE-13809.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10233 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_repair
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_table_nonprintable
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testPartitionsCheck
org.apache.hadoop.hive.ql.metadata.TestHiveMetaStoreChecker.testTableCheck
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/148/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/148/console
Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-148/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12811158 - PreCommit-HIVE-MASTER-Build

> Hybrid Grace Hash Join memory usage estimation didn't take into account the bloom filter size
> ---------------------------------------------------------------------------------------------
>
>                 Key: HIVE-13809
>                 URL: https://issues.apache.org/jira/browse/HIVE-13809
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive
>    Affects Versions: 2.0.0, 2.1.0
>            Reporter: Wei Zheng
>            Assignee: Wei Zheng
>         Attachments: HIVE-13809.1.patch
>
>
> Memory estimation is important during hash table loading, because we need to make the decision of whether to load the next hash partition in memory or spill it. If the assumption is there's enough memory but it turns out not the case, we will run into OOM problem.
> Currently hybrid grace hash join memory usage estimation didn't take into account the bloom filter size. In large test cases (TB scale) the bloom filter grows as big as hundreds of MB, big enough to cause estimation error.
> The solution is to count in the bloom filter size into memory estimation.
> Another issue this patch will fix is possible NPE due to object cache reuse during hybrid grace hash join.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)