Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id CA4FB200B17 for ; Tue, 7 Jun 2016 07:06:09 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id C8F68160A55; Tue, 7 Jun 2016 05:06:09 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1BA97160A24 for ; Tue, 7 Jun 2016 07:06:08 +0200 (CEST) Received: (qmail 58079 invoked by uid 500); 7 Jun 2016 05:06:08 -0000 Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@impala.incubator.apache.org Delivered-To: mailing list dev@impala.incubator.apache.org Received: (qmail 58068 invoked by uid 99); 7 Jun 2016 05:06:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 07 Jun 2016 05:06:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 875BE1803EA for ; Tue, 7 Jun 2016 05:06:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.362 X-Spam-Level: X-Spam-Status: No, score=0.362 tagged_above=-999 required=6.31 tests=[RDNS_DYNAMIC=0.363, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id tBdZqVx6KPDC for ; Tue, 7 Jun 2016 05:06:05 +0000 (UTC) Received: from ip-10-146-233-104.ec2.internal (ec2-75-101-130-251.compute-1.amazonaws.com [75.101.130.251]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id A8E4F5F306 for ; Tue, 7 Jun 2016 05:06:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ip-10-146-233-104.ec2.internal (8.14.4/8.14.4) with ESMTP id u57563Tv016749; Tue, 7 Jun 2016 05:06:03 GMT Message-Id: <201606070506.u57563Tv016749@ip-10-146-233-104.ec2.internal> Date: Tue, 7 Jun 2016 05:06:03 +0000 From: "Tim Armstrong (Code Review)" To: impala-cr@cloudera.com, dev@impala.incubator.apache.org CC: Matthew Jacobs , Dan Hecht Reply-To: tarmstrong@cloudera.com X-Gerrit-MessageType: merged Subject: =?UTF-8?Q?=5BImpala-CR=5D=28cdh5-trunk=29_IMPALA-3670=3A_fix_sorter_buffer_mgmt_bugs=0A?= X-Gerrit-Change-Id: Ia1a0ddffa0a5b157ab86a376b7b7360a923698d6 X-Gerrit-ChangeURL: X-Gerrit-Commit: 42326684c44f5d9dede8089250d842f1c48cbec2 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.2 archived-at: Tue, 07 Jun 2016 05:06:10 -0000 Tim Armstrong has submitted this change and it was merged. Change subject: IMPALA-3670: fix sorter buffer mgmt bugs ...................................................................... IMPALA-3670: fix sorter buffer mgmt bugs Also make test_scratch_disk.py more deterministic, by using max_block_mgr_memory, which doesn't include scanner memory. The fixed test_scratch_disk.py exercises the other sorter bugs that occurs when scratch cannot be written. Testing: Added a test that does a sort with various memory limits and consumes the whole output of the sorter (we have many tests of sorts with limits but limited coverage of sorts without limits). Ran an exhaustive test run before posting for review. This added test reproduced one of the sorter bugs, where var-len blocks were not always attached to the output batch. The other test was reproduced by the test change in IMPALA-3669: test_scratch_disk fix. Change-Id: Ia1a0ddffa0a5b157ab86a376b7b7360a923698d6 Reviewed-on: http://gerrit.cloudera.org:8080/3315 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M be/src/runtime/sorter.cc M tests/query_test/test_sort.py 2 files changed, 45 insertions(+), 25 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/3315 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ia1a0ddffa0a5b157ab86a376b7b7360a923698d6 Gerrit-PatchSet: 7 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Matthew Jacobs Gerrit-Reviewer: Tim Armstrong