impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-5073: Use mmap() instead of malloc() for buffer pool
Date Fri, 07 Apr 2017 23:12:15 GMT
Tim Armstrong has uploaded a new patch set (#5).

Change subject: IMPALA-5073: Use mmap() instead of malloc() for buffer pool

IMPALA-5073: Use mmap() instead of malloc() for buffer pool

Allocate with mmap instead of TCMalloc to give more control over memory
usage. Also tell Linux to back larger buffers with huge pages when
possible to reduce TLB pressure. The main complication is that memory
returned by mmap() is not necessarily aligned to a huge page boundary,
so we need to "fix up" the mapping ourselves.

Adds additional memory metrics, since we previously relied on the
assumption that all memory was allocated through TCMalloc. tracks the total across the buffer pool and
TCMalloc. When the buffer pool is not present, they just report
the TCMalloc values.

ASAN still uses malloc() because it doesn't instrument mmap() by
design - see

Added some unit tests to test edge cases. Many pre-existing tests also
exercise the modified code.

Change-Id: Ifbc748f74adcbbdcfa45f3ec7df98284925acbd6
M be/src/catalog/
M be/src/runtime/bufferpool/
M be/src/runtime/bufferpool/buffer-allocator.h
M be/src/runtime/bufferpool/
M be/src/runtime/bufferpool/buffer-pool.h
M be/src/runtime/bufferpool/
M be/src/runtime/bufferpool/system-allocator.h
M be/src/runtime/
M be/src/statestore/
M be/src/util/asan.h
M be/src/util/
M be/src/util/memory-metrics.h
M be/src/util/
M be/src/util/metrics.h
M common/thrift/
M common/thrift/metrics.json
16 files changed, 363 insertions(+), 36 deletions(-)

  git pull ssh:// refs/changes/74/6474/5
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifbc748f74adcbbdcfa45f3ec7df98284925acbd6
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <>

View raw message