impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Behm (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-2789: More compact mem layout with null bits at the end.
Date Tue, 11 Oct 2016 20:02:17 GMT
Alex Behm has uploaded a new patch set (#3).

Change subject: IMPALA-2789: More compact mem layout with null bits at the end.

IMPALA-2789: More compact mem layout with null bits at the end.

The main motivation of this change is to pave the way for full
memory layout compatibility between Impala and Kudu to eventually
enable zero-copy scans. This patch is a only first step towards
that goal.

New Memory Layout
Slots are placed in descending order by size with trailing bytes to
store null flags. Null flags are omitted for non-nullable slots. There
is no padding between tuples when stored back-to-back in a row batch.

Example: select bool_col, int_col, string_col, smallint_col
         from functional.alltypes
Slots:   string_col|int_col|smallint_col|bool_col|null_byte
Offsets: 0          16      20           22       23

The main change is to move the null indicators to the end of tuples.
The new memory layout is fully packed with no padding in between
slots or tuples.

Our standard cluster perf tests showed no significant difference in
query response times as well as consumed cycles, and a slight
reduction in peak memory consumption.

An exhaustive test run passed. Ran a few select tests like TPC-H/DS
with ASAN locally.

Change-Id: Ib6510c75d841bddafa6638f1bd2ac6731a7053f6
M be/src/benchmarks/
M be/src/codegen/
M be/src/codegen/llvm-codegen.h
M be/src/exec/
M be/src/exec/hdfs-scanner.h
M be/src/exec/
M be/src/exec/kudu-scanner.h
M be/src/exec/
M be/src/exec/kudu-testutil.h
M be/src/exec/
M be/src/exec/
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/descriptors.h
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/tuple.h
M be/src/service/
M be/src/service/frontend.h
M be/src/testutil/
M be/src/testutil/desc-tbl-builder.h
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/analysis/
M fe/src/main/java/org/apache/impala/analysis/
M fe/src/main/java/org/apache/impala/service/
M fe/src/test/java/org/apache/impala/analysis/
28 files changed, 406 insertions(+), 315 deletions(-)

  git pull ssh:// refs/changes/73/4673/3
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib6510c75d841bddafa6638f1bd2ac6731a7053f6
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <>
Gerrit-Reviewer: Alex Behm <>
Gerrit-Reviewer: Tim Armstrong <>

View raw message