impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henry Robinson (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-4856: Port data stream service to KRPC
Date Thu, 22 Jun 2017 00:00:44 GMT
Henry Robinson has uploaded a new patch set (#4).

Change subject: IMPALA-4856: Port data stream service to KRPC

IMPALA-4856: Port data stream service to KRPC

This patch ports the data-flow parts of ImpalaInternalService to KRPC.

* ImpalaInternalService is split into two services. The first,
  ImpalaInternalService, deals with control messages for plan fragment
  instance execution, cancellation and reporting, and remains
  implemented in Thrift for now. The second, DataStreamService, handles
  large-payload RPCs for transmitting runtime filters and row batches
  between hosts.

* In the DataStreamService, all RPCs use 'native' protobuf. The
  DataStreamService starts on the port previously reserved for the
  StatestoreSubscriberService (which is also a KRPC service), to avoid
  having to configure another port when starting Impala. When the
  ImpalaInternalService is ported to KRPC, all services will run on one

* To support needing to address two different backend services, a data
  service port has been added to TBackendDescriptor.

* This patch adds support for asynchronous RPCs to the RpcMgr and Rpc
  classes. Previously, Impala used fixed size thread pools + synchronous
  RPCs to achieve some parallelism for 'broadcast' RPCs like filter
  propagation, or a dedicated per-sender+receiver pair thread on the
  sender side in the DataStreamSender case. In this patch, the
  PublishFilter() and TransmitData() RPCs are sent asynchronously using
  KRPC's thread pools.

* The TransmitData() protocol has changed to adapt to asynchronous
  RPCs. The full details are in data-stream-mgr.h.

* As a result, DataStreamSender no longer creates a
  thread-per-connection on the sender side.

* Both tuple transmission and runtime filter publication use sidecars to
  minimise the number of copies and serialization steps required.

* Also include a fix for KUDU-2011 that properly allows sidecars to be
  shared between KRPC and the RPC caller (fixing IMPALA-5093, a
  corruption bug).

* A large portion of this patch is the replacement of TRowBatch with its
  Protobuf equivalent, RowBatchPb. The replacement is a literal port of
  the data structure, and row-batch-test, row-batch-list-test and
  row-batch-serialize-benchmark continue to execute without major logic

* Simplify FindRecvr() logic in DataStreamManager. No-longer need to
  handle blocking sender-side, so no need for complex promise-based
  machinery. Instead, all senders with no receiver are added to a
  per-receiver list, which is processed when the receiver arrives. If it
  does not arrive promptly, the DataStreamManager cleans them up after

* This patch also begins a clean-up of how ImpalaServer instances are
  created (by removing CreateImpalaServer), and clarifying the
  relationship between ExecEnv and ImpalaServer. ImpalaServer now
  follows the standard construct->Init()->Start()->Join() lifecycle that
  we use for other services.

* Ensure that all addresses used for KRPCs are fully resolved, avoiding
  the need to resolve them for each RPC.


* New tests added to rpc-mgr-test.


* Re-enable throughput and latency measurements per data-stream sender
  when that information is exposed from KRPC (KUDU-1738).

* TLS and Kerberos are still not supported by KRPC in this patch.

Change-Id: Ia66704be7a0a8162bb85556d07b583ec756c584b
M .clang-format
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/benchmarks/
M be/src/benchmarks/
M be/src/common/
M be/src/common/
M be/src/common/status.h
M be/src/exprs/
M be/src/kudu/rpc/
M be/src/kudu/rpc/rpc_sidecar.h
M be/src/rpc/CMakeLists.txt
M be/src/rpc/TAcceptQueueServer.cpp
M be/src/rpc/common.proto
M be/src/rpc/
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc-mgr.inline.h
M be/src/rpc/rpc.h
M be/src/runtime/backend-client.h
M be/src/runtime/client-cache-types.h
M be/src/runtime/
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator-filter-state.h
M be/src/runtime/
M be/src/runtime/coordinator.h
M be/src/runtime/
M be/src/runtime/data-stream-mgr.h
M be/src/runtime/
M be/src/runtime/data-stream-recvr.h
M be/src/runtime/
M be/src/runtime/data-stream-sender.h
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/exec-env.h
M be/src/runtime/
M be/src/runtime/fragment-instance-state.h
M be/src/runtime/
M be/src/runtime/query-state.h
M be/src/runtime/
M be/src/runtime/
M be/src/runtime/row-batch.h
M be/src/runtime/
M be/src/runtime/runtime-filter-bank.h
M be/src/scheduling/
M be/src/scheduling/
M be/src/scheduling/backend-config.h
M be/src/scheduling/query-schedule.h
M be/src/scheduling/
M be/src/scheduling/scheduler-test-util.h
M be/src/scheduling/
M be/src/scheduling/scheduler.h
M be/src/service/CMakeLists.txt
A be/src/service/data_stream_service.proto
M be/src/service/
M be/src/service/
M be/src/service/impala-internal-service.h
M be/src/service/
M be/src/service/impala-server.h
M be/src/service/
M be/src/service/
M be/src/statestore/
M be/src/statestore/
M be/src/statestore/
M be/src/testutil/fault-injection-util.h
M be/src/testutil/
M be/src/testutil/in-process-servers.h
M be/src/util/blocking-queue.h
M be/src/util/
M be/src/util/
M be/src/util/bloom-filter.h
M be/src/util/
M be/src/util/
M be/src/util/network-util.h
M be/src/util/thread-pool.h
M bin/
M common/thrift/CMakeLists.txt
M common/thrift/ImpalaInternalService.thrift
M common/thrift/Results.thrift
M common/thrift/StatestoreService.thrift
M common/thrift/
M tests/custom_cluster/
M tests/custom_cluster/
M tests/custom_cluster/
M tests/query_test/
M tests/stress/
85 files changed, 2,771 insertions(+), 1,971 deletions(-)

  git pull ssh:// refs/changes/03/7103/4
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia66704be7a0a8162bb85556d07b583ec756c584b
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson <>
Gerrit-Reviewer: Henry Robinson <>
Gerrit-Reviewer: Michael Ho <>
Gerrit-Reviewer: Sailesh Mukil <>

View raw message