impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-4674: Part 1: port BufferedTupleStream to BufferPool
Date Fri, 27 Jan 2017 19:38:04 GMT
Tim Armstrong has uploaded a new patch set (#4).

Change subject: IMPALA-4674: Part 1: port BufferedTupleStream to BufferPool

IMPALA-4674: Part 1: port BufferedTupleStream to BufferPool

Add a copy of BufferedTupleStream that allocates memory from BufferPool.
This will replace the original implementation when IMPALA-3200 is

The major changes are:
* Terminology is updated from "blocks" to "pages"
* No small buffer support is needed (hooray!).
* BufferedTupleStream needs to do its own tracking of # of rows per
  page, etc instead of relying on BufferedBlockMgr to do it. A
  wrapper around PageHandle is used.
* Profile counters (unpin, pin, get new block time) are removed -
  similar counters in the BufferPool client are more useful.

The logic for serialising and deserialising rows has not changed,
since that is implemented in terms of read and write pointers into
the buffers.

Testing this required some further changes. TestEnv was refactored
so it can set up either BufferPool or BufferedBlockMgr. Some
BufferPool-related state is added to ExecEnv, QueryState and
RuntimeState, but is only created for backend tests that explicitly
create a BufferPool.

The following is left for future work:
* IMPALA-3808 (large row support) is not added. I've added
  TODOs to the code to places that will require changes.
* IMPALA-4179 (remove MarkNeedsDeepCopy()) is not fixed, since
  it requires global changes to operators that accumulate memory.

All of the BufferedTupleStream unit tests are ported to the new
implementation, except for ones specifically testing the small
buffer functionality.

Change-Id: I7bb47a818564ac19a6be4ca88db0578f4ea0b709
M be/src/exec/
M be/src/runtime/CMakeLists.txt
M be/src/runtime/
M be/src/runtime/
A be/src/runtime/
A be/src/runtime/
A be/src/runtime/buffered-tuple-stream-v2.h
A be/src/runtime/buffered-tuple-stream-v2.inline.h
M be/src/runtime/bufferpool/
M be/src/runtime/
M be/src/runtime/exec-env.h
M be/src/runtime/
M be/src/runtime/query-state.h
M be/src/runtime/
M be/src/runtime/row-batch.h
M be/src/runtime/
M be/src/runtime/runtime-state.h
M be/src/runtime/
M be/src/runtime/test-env.h
M be/src/runtime/
20 files changed, 2,942 insertions(+), 99 deletions(-)

  git pull ssh:// refs/changes/11/5811/4
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7bb47a818564ac19a6be4ca88db0578f4ea0b709
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <>
Gerrit-Reviewer: Dan Hecht <>

View raw message