impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Tauber-Marshall (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-5167: Reduce the number of Kudu clients created
Date Tue, 09 May 2017 21:35:28 GMT
Thomas Tauber-Marshall has uploaded a new patch set (#3).

Change subject: IMPALA-5167: Reduce the number of Kudu clients created

IMPALA-5167: Reduce the number of Kudu clients created

Creating Kudu clients is very expensive as each will fetch
metadata from the Kudu master, so we should minimize the
number of Kudu clients that get created.

This patch stores a map from Kudu master addressed to Kudu
clients in the ExecEnv to be used across the BE for all
queries, and a map in KuduUtil to be used acress the FE
and catalog for all queries.

This relies on changes on the Kudu side that clear
non-covered range entries from the client's cache on
table open (fdc022fe6231af20e307012d98c35b16cbfa7b33
and d07ecd6ded01201c912d2e336611a6a941f48d98)

- Ran a stress test locally: scan of a Kudu table, 100
  concurrent queries, load on the Kudu master was reduced
  signficantly, from ~25% cpu to ~4% cpu.
  TODO: perf testing on a cluster.
- Ran the Kudu e2e tests.
- Manually ran a test with concurrent INSERTs and
  'ALTER TABLE ADD PARTITION' (which is affected by the
  Kudu side change mentiond above) and verified

Change-Id: I6b0c12a256c33e8ef32315b3736cae2dea2ae705
M be/src/exec/
M be/src/exec/
M be/src/exec/kudu-table-sink.h
M be/src/exprs/
M be/src/runtime/
M be/src/runtime/exec-env.h
M be/src/runtime/
M be/src/runtime/query-state.h
M fe/src/main/java/org/apache/impala/catalog/
M fe/src/main/java/org/apache/impala/planner/
M fe/src/main/java/org/apache/impala/service/
M fe/src/main/java/org/apache/impala/util/
12 files changed, 116 insertions(+), 91 deletions(-)

  git pull ssh:// refs/changes/92/6792/3
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b0c12a256c33e8ef32315b3736cae2dea2ae705
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <>
Gerrit-Reviewer: Marcel Kornacker <>
Gerrit-Reviewer: Matthew Jacobs <>
Gerrit-Reviewer: Thomas Tauber-Marshall <>

View raw message