Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 7AAF3200BAC for ; Wed, 26 Oct 2016 22:46:47 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 793B3160AEE; Wed, 26 Oct 2016 20:46:47 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id BF3CB160AE1 for ; Wed, 26 Oct 2016 22:46:46 +0200 (CEST) Received: (qmail 16230 invoked by uid 500); 26 Oct 2016 20:46:46 -0000 Mailing-List: contact reviews-help@impala.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@impala.incubator.apache.org Received: (qmail 16219 invoked by uid 99); 26 Oct 2016 20:46:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 26 Oct 2016 20:46:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 368EFC7873 for ; Wed, 26 Oct 2016 20:46:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.362 X-Spam-Level: X-Spam-Status: No, score=0.362 tagged_above=-999 required=6.31 tests=[RDNS_DYNAMIC=0.363, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id uohR3dmeEdkl for ; Wed, 26 Oct 2016 20:46:42 +0000 (UTC) Received: from ip-10-146-233-104.ec2.internal (ec2-75-101-130-251.compute-1.amazonaws.com [75.101.130.251]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 3A8565F240 for ; Wed, 26 Oct 2016 20:46:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by ip-10-146-233-104.ec2.internal (8.14.4/8.14.4) with ESMTP id u9QKkaqo025840; Wed, 26 Oct 2016 20:46:36 GMT Message-Id: <201610262046.u9QKkaqo025840@ip-10-146-233-104.ec2.internal> Date: Wed, 26 Oct 2016 20:46:36 +0000 From: "Tim Armstrong (Code Review)" To: impala-cr@cloudera.com, reviews@impala.incubator.apache.org CC: Dan Hecht Reply-To: tarmstrong@cloudera.com X-Gerrit-MessageType: newpatchset Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-3202=3A_DiskIoMgr_improvements_for_new_buffer_pool=0A?= X-Gerrit-Change-Id: I913fbb8ae6c242f1585992eb2a622509344dccec X-Gerrit-ChangeURL: X-Gerrit-Commit: d7075231e4ce4abf12abdd671af7857127232d53 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.2 archived-at: Wed, 26 Oct 2016 20:46:47 -0000 Tim Armstrong has uploaded a new patch set (#4). Change subject: IMPALA-3202: DiskIoMgr improvements for new buffer pool ...................................................................... IMPALA-3202: DiskIoMgr improvements for new buffer pool The main goal of this patch is to add support to the DiskIoMgr to read into client-provided buffers, instead of the DiskIoMgr allocating intermediate buffers. This is important for the new buffer pool because we want it to entirely manage its own memory, rather than delegating part of its memory management to the DiskIoMgr. Also do some cleanup: * Consolidate some correlated ScanRange parameters into a parameter struct. * Remove the "untracked" buffers mem tracker, which is no longer necessary. * Change the buffer type in DiskIoMgr to use uint8_t* to be consistent with the rest of Impala. Testing: Added a unit test. We also get coverage from the BufferedBlockMgr unit tests, any spilling tests and the Parquet tests with large footers. Change-Id: I913fbb8ae6c242f1585992eb2a622509344dccec --- M be/src/exec/base-sequence-scanner.cc M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-text-scanner.cc M be/src/exec/scanner-context.cc M be/src/runtime/buffered-block-mgr.cc M be/src/runtime/disk-io-mgr-internal.h M be/src/runtime/disk-io-mgr-scan-range.cc M be/src/runtime/disk-io-mgr-stress-test.cc M be/src/runtime/disk-io-mgr-stress.cc M be/src/runtime/disk-io-mgr-stress.h M be/src/runtime/disk-io-mgr-test.cc M be/src/runtime/disk-io-mgr.cc M be/src/runtime/disk-io-mgr.h 15 files changed, 404 insertions(+), 224 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/4631/4 -- To view, visit http://gerrit.cloudera.org:8080/4631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I913fbb8ae6c242f1585992eb2a622509344dccec Gerrit-PatchSet: 4 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong