impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Volker (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-6193: Track memory of incoming data streams
Date Fri, 22 Dec 2017 18:02:42 GMT
Lars Volker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8914


Change subject: IMPALA-6193: Track memory of incoming data streams
......................................................................

IMPALA-6193: Track memory of incoming data streams

This changes adds memory tracking to incoming transmit data RPCs when
using KRPC. We track memory inside the service pools until it is handed
over to the stream manager. There we track it in a global tracker until
a receiver registers and takes over the early sender RPCs. Inside the
receiver memory for deferred RPCs is tracked against the instance
tracker until we unpack the batches and add them to the row batch queue.

I tested this change manually by running TPCH and TPCDS on my local dev
machine and on a private cluster.

TODO: more testing

Change-Id: I2df1204d2483313a8a18e5e3be6cec9e402614c4
---
M be/src/rpc/impala-service-pool.cc
M be/src/rpc/impala-service-pool.h
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/runtime/exec-env.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-mgr.h
A be/src/runtime/krpc-data-stream-mgr.inline.h
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-recvr.h
M be/src/util/memory-metrics.h
M common/protobuf/data_stream_service.proto
12 files changed, 289 insertions(+), 156 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/8914/1
-- 
To view, visit http://gerrit.cloudera.org:8080/8914
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2df1204d2483313a8a18e5e3be6cec9e402614c4
Gerrit-Change-Number: 8914
Gerrit-PatchSet: 1
Gerrit-Owner: Lars Volker <lv@cloudera.com>

Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message