impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (Code Review)" <>
Subject [Toolchain-CR] IMPALA-3282: TBinaryProtocol buffers consume unbounded memory
Date Tue, 24 May 2016 21:21:40 GMT
Tim Armstrong has uploaded a new patch set (#4).

Change subject: IMPALA-3282: TBinaryProtocol buffers consume unbounded memory

IMPALA-3282: TBinaryProtocol buffers consume unbounded memory

This patch removes the builtin string buffer in TBinaryProtocol. It was
used to buffer large strings being read via thrift. The buffer is never
shrunk, so each thrift connection ends up consuming memory equal to the
largest string that was received by the connection. This can result in a
lot of memory, given a large number of connections and large strings.

The string buffer is actually unnecessary, since the buffer contents are
always copied into the string, so the buffer simply adds an extra copy.
This may not be a big win, since string::resize() implicitly does a

The fix is to resize the string then write directly to the string's memory.
The C++11 standard guarantees that a std::string's memory is contiguous.

Did a local build and ran a local stress test with heap profiling.
Inspected heap profile to make sure that we did not have large
allocations from TBinaryProtocol::readStringBody() sitting
around when Impala is idle.

Change-Id: I58bdc899e604bde2f5b2d5360f79537ce25715ee
A source/thrift/thrift-0.9.0-patches/0008-IMPALA-3582-string-buffer-memory.patch
2 files changed, 70 insertions(+), 0 deletions(-)

  git pull ssh:// refs/changes/01/3201/4
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I58bdc899e604bde2f5b2d5360f79537ce25715ee
Gerrit-PatchSet: 4
Gerrit-Project: Toolchain
Gerrit-Branch: master
Gerrit-Owner: Tim Armstrong <>
Gerrit-Reviewer: Dan Hecht <>
Gerrit-Reviewer: Dimitris Tsirogiannis <>
Gerrit-Reviewer: Huaisi Xu <>

View raw message