Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 355AF10C5A for ; Fri, 14 Mar 2014 06:10:47 +0000 (UTC) Received: (qmail 94288 invoked by uid 500); 14 Mar 2014 06:10:45 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 94201 invoked by uid 500); 14 Mar 2014 06:10:44 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 94185 invoked by uid 500); 14 Mar 2014 06:10:44 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 94179 invoked by uid 99); 14 Mar 2014 06:10:43 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Mar 2014 06:10:43 +0000 Date: Fri, 14 Mar 2014 06:10:43 +0000 (UTC) From: "Hive QA (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-6518) Add a GC canary to the VectorGroupByOperator to flush whenever a GC is triggered MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-6518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13934667#comment-13934667 ] Hive QA commented on HIVE-6518: ------------------------------- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12634401/HIVE-6518.3.patch {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 5389 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_auto_sortmerge_join_16 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucketmapjoin6 {noformat} Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1769/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/1769/console Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12634401 > Add a GC canary to the VectorGroupByOperator to flush whenever a GC is triggered > -------------------------------------------------------------------------------- > > Key: HIVE-6518 > URL: https://issues.apache.org/jira/browse/HIVE-6518 > Project: Hive > Issue Type: Bug > Components: Query Processor > Affects Versions: 0.13.0 > Reporter: Gopal V > Assignee: Gopal V > Priority: Minor > Attachments: HIVE-6518.1-tez.patch, HIVE-6518.2-tez.patch, HIVE-6518.2.patch, HIVE-6518.3.patch > > > The current VectorGroupByOperator implementation flushes the in-memory hashes when the maximum entries or fraction of memory is hit. > This works for most cases, but there are some corner cases where we hit GC ovehead limits or heap size limits before either of those conditions are reached due to the rest of the pipeline. > This patch adds a SoftReference as a GC canary. If the soft reference is dead, then a full GC pass happened sometime in the near past & the aggregation hashtables should be flushed immediately before another full GC is triggered. -- This message was sent by Atlassian JIRA (v6.2#6252)