impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Jacobs (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-4478: Initial Kudu client mem tracking for sink
Date Sat, 19 Nov 2016 21:11:53 GMT
Matthew Jacobs has uploaded a new change for review.

Change subject: IMPALA-4478: Initial Kudu client mem tracking for sink

IMPALA-4478: Initial Kudu client mem tracking for sink

The Kudu client allocates memory which is not tracked by
Impala. There are several sources, but the most significant
is the memory allocated by the KuduSession on the write
path. This can be >100MB, so it is important to track to
avoid OOM.

Moving forward, we should have a better way to track Kudu
client memory, but for now we must at least handle this
potentially problematic case.

This changes the KuduTableSink to consume 200MB which should
be enough for the 100MB write mutation buffer as well as
100MB worth of errors buffered in the client before Impala
takes ownership of them (and deletes them). This is left as
a flag because it may turn out to be too high for some users
and too low for others. When we have better support from
Kudu (including KUDU-1752), we should simplify this.

TODO: Handle DML w/ small or known resource requirements
(e.g. VALUES specified or query has LIMIT) specially to
avoid over-consumption.

Testing: Have verified acceptable behavior in the stress
test with a simple workload containing DML statements of
moderate cardinality.

Change-Id: I47f17a81e4362ab490019382fedc66c25f07080a
M be/src/exec/
1 file changed, 32 insertions(+), 1 deletion(-)

  git pull ssh:// refs/changes/52/5152/1
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newchange
Gerrit-Change-Id: I47f17a81e4362ab490019382fedc66c25f07080a
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs <>

View raw message