impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Behm (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-3905: HdfsScanner::GetNext() for Avro, RC, and Seq scans.
Date Thu, 13 Apr 2017 00:16:22 GMT
Alex Behm has uploaded a new patch set (#2).

Change subject: IMPALA-3905: HdfsScanner::GetNext() for Avro, RC, and Seq scans.

IMPALA-3905: HdfsScanner::GetNext() for Avro, RC, and Seq scans.

Implements HdfsScanner::GetNext() for the Avro, RC File, and
Sequence File scanners. Changes ProcessSplit() to repeatedly call
GetNext() to share the core scanning code between the legacy
ProcessSplit() interface (ProcessSpit()) and the new GetNext()

Summary of changes:
- Slightly change code flow for initial scan range that
  only parses the file header. The new code sets
  'only_parsing_header_' in Open() and then honors
  that flag in GetNextInternal(). Before, all the logic
  was inside ProcessSpit().
- Replace 'finished_' with 'eos_'.
- Add a RowBatch parameter to various functions.
- Change Close() to free all resources when a nullptr
  RowBatch is passed.

- Exhaustive tests passed on debug
- Core tests passed on asan
- TODO: Perf testing on cluster

Change-Id: Ie18f57b0d3fe0052a8ccd361b6a5fcdf979d0669
M be/src/exec/
M be/src/exec/base-sequence-scanner.h
M be/src/exec/
M be/src/exec/
M be/src/exec/hdfs-avro-scanner.h
M be/src/exec/
M be/src/exec/
M be/src/exec/hdfs-rcfile-scanner.h
M be/src/exec/
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/
M be/src/exec/hdfs-scan-node.h
M be/src/exec/
M be/src/exec/hdfs-scanner.h
M be/src/exec/
M be/src/exec/hdfs-sequence-scanner.h
M be/src/exec/
M be/src/exec/hdfs-text-scanner.h
M be/src/util/
M testdata/workloads/functional-query/queries/DataErrorsTest/avro-errors.test
20 files changed, 527 insertions(+), 604 deletions(-)

  git pull ssh:// refs/changes/27/6527/2
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie18f57b0d3fe0052a8ccd361b6a5fcdf979d0669
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <>
Gerrit-Reviewer: Marcel Kornacker <>

View raw message