Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
Date: Wed, 24 Aug 2016 18:05:17 +0000
From: "Alex Behm (Code Review)" <gerrit@cloudera.org>
To: impala-cr@cloudera.com, dev@impala.incubator.apache.org
Message-ID: <gerrit.1472061917000.I98cc7f970e1575dd83875609985e1877ada3d5e0@gerrit.cloudera.org>
Reply-To: alex.behm@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-ASF-CR=5D_IMPALA-3905=3A_Add_single-threaded_scan_node=2E=0A?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.2
archived-at: Wed, 24 Aug 2016 18:05:26 -0000

Alex Behm has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/4113

Change subject: IMPALA-3905: Add single-threaded scan node.
......................................................................

IMPALA-3905: Add single-threaded scan node.

Adds a new single-threaded scan node HdfsScanNodeMt that
materializes tuples in the thread calling GetNext().
The new scan node uses the HdfsScanner::GetNext() interface,
which currently is only implemented for Parquet.
As before, I/O is performed asynchronously via the I/O manager.

The new scan node is enabled if the mt_dop query option is
set to a value greater than 1. Otherwise, the existing
multi-threaded scan node is used.

The changes are mostly a refactoring of the existing multi-threaded
scan node to separate out the common code between the existing
multi-threaded scan node and the new single-threaded one.

Summary of changes:
- Move code from hdfs-scan-node.h/cc into a new hdfs-scan-node-base.h/cc.
- Add new single-threaded scan node in hdfs-scan-node-mt.h/cc.
- Both scan nodes inherit from HdfsScanNodeBase.
- Rework the allocation of templates tuples such that the memory is drawn
  from a new mem pool in the scanners, and that each scanner clones the
  partition exprs contexts. Before, the memory was taken from the parent
  scan node's mem pool, and there was only one instance of the partition
  exprs contexts. Their access was protected under a lock, however, not
  in all instances, so their use was not always obviously correct.
  The change in this patch makes thread safety obvious and helps move
  a lock into the multi-threaded scan node which would otherwise have
  to remain in the HdfsScanNodeBase class.
- ScannerContext::cancelled() is used for synchronizing threads in
  multi-threaded scans. Changed it to always return true when used
  in the context of the new single-threaded scan node.
- Simplify a couple of loops with C++11 for-each.

There are currently no automated tests that exercise the
single-threaded scan node.

Testing: Passed debug/exhaustive and asan/core builds on HDFS.

Change-Id: I98cc7f970e1575dd83875609985e1877ada3d5e0
---
M be/src/exec/CMakeLists.txt
M be/src/exec/base-sequence-scanner.cc
M be/src/exec/base-sequence-scanner.h
M be/src/exec/exec-node.cc
M be/src/exec/hdfs-avro-scanner.cc
M be/src/exec/hdfs-avro-scanner.h
M be/src/exec/hdfs-lzo-text-scanner.cc
M be/src/exec/hdfs-lzo-text-scanner.h
M be/src/exec/hdfs-parquet-scanner.cc
M be/src/exec/hdfs-parquet-scanner.h
M be/src/exec/hdfs-rcfile-scanner.cc
M be/src/exec/hdfs-rcfile-scanner.h
A be/src/exec/hdfs-scan-node-base.cc
A be/src/exec/hdfs-scan-node-base.h
A be/src/exec/hdfs-scan-node-mt.cc
A be/src/exec/hdfs-scan-node-mt.h
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/hdfs-scan-node.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-scanner.h
M be/src/exec/hdfs-sequence-scanner.cc
M be/src/exec/hdfs-sequence-scanner.h
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/hdfs-text-scanner.h
M be/src/exec/scanner-context.cc
M be/src/exec/scanner-context.h
M be/src/exprs/expr-context.h
M be/src/runtime/tuple.h
28 files changed, 1,712 insertions(+), 1,384 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/4113/1
-- 
To view, visit http://gerrit.cloudera.org:8080/4113
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I98cc7f970e1575dd83875609985e1877ada3d5e0
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <alex.behm@cloudera.com>