impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Impala Public Jenkins (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-3200: Implement suballocator for splitting buffers
Date Thu, 15 Dec 2016 08:01:22 GMT
Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-3200: Implement suballocator for splitting buffers

IMPALA-3200: Implement suballocator for splitting buffers

This is useful for situations like hash tables, where we want to
make multiple non-spillable allocations of variable size from buffer
pool memory and not incur the overhead of interacting with the global
buffer pool. The allocator subdivides buffers to service allocations
and uses a buddy allocation algorithm to merge freed allocations into
larger chunks. This helps avoid external fragmentation and is quite
effective at reusing memory given the typical doubling allocation
patterns of hash tables in partitioned aggs and joins.

The allocator has fairly robust internal consistency checks via
assertions and unique_ptrs. Includes a unit test that exercises
various allocation patterns.

I tested porting hash tables over to allocate memory using an earlier
version of the suballocator, which worked well (was able to run a
wide range of queries successfully with good performance).

Change-Id: I8bfe0e429f67ad273f7c7d0816703a9e6c3da788
Reviewed-by: Tim Armstrong <>
Tested-by: Impala Public Jenkins
M be/src/common/names.h
M be/src/exec/hash-table.h
M be/src/runtime/bufferpool/CMakeLists.txt
A be/src/runtime/bufferpool/
A be/src/runtime/bufferpool/
A be/src/runtime/bufferpool/suballocator.h
6 files changed, 857 insertions(+), 0 deletions(-)

  Impala Public Jenkins: Verified
  Tim Armstrong: Looks good to me, approved

To view, visit
To unsubscribe, visit

Gerrit-MessageType: merged
Gerrit-Change-Id: I8bfe0e429f67ad273f7c7d0816703a9e6c3da788
Gerrit-PatchSet: 20
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <>
Gerrit-Reviewer: Dan Hecht <>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple <>
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Michael Ho <>
Gerrit-Reviewer: Tim Armstrong <>

View raw message