impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pooja Nilangekar (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] Build a ConcatenatedStreams wrapper for ScannerContext::Stream
Date Thu, 27 Jul 2017 00:14:10 GMT
Pooja Nilangekar has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/7513

Change subject: Build a ConcatenatedStreams wrapper for ScannerContext::Stream
......................................................................

Build a ConcatenatedStreams wrapper for ScannerContext::Stream

The ConcatenatedStreams class keeps track of a vector of streams which
can be utilized by the BaseScalarColumnReader to read transparently
from a list of streams. The ConcatenatedStreams would hide the
different ScanRanges from the caller and provide the abstraction of
a single stream. The class uses a list of streams and a index to point
to the current active stream. The class moves to the next stream once
the current stream reaches its end of stream.
TODO: Currently, the ConcatenatedStreams don't support reads which span
across multiple streams. Thus the maximum bytes you can read at any
given instance is dictated by the number of bytes remaining in the
current active stream.

Testing: The existing test_scanners.py test was used to validate the
correctness of the ConcatenatedStreams.

Change-Id: I026f7a005ba4d6fc412ca0884f4fa0459da9c885
---
M be/src/exec/hdfs-parquet-scanner.cc
M be/src/exec/parquet-column-readers.h
M be/src/exec/scanner-context.cc
M be/src/exec/scanner-context.h
4 files changed, 125 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/7513/1
-- 
To view, visit http://gerrit.cloudera.org:8080/7513
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I026f7a005ba4d6fc412ca0884f4fa0459da9c885
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Pooja Nilangekar <pooja.nilangekar@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv@cloudera.com>

Mime
View raw message