impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henry Robinson (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4856: Port ImpalaInternalService to KRPC
Date Fri, 03 Feb 2017 01:24:01 GMT
Henry Robinson has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/5888

Change subject: IMPALA-4856: Port ImpalaInternalService to KRPC
......................................................................

IMPALA-4856: Port ImpalaInternalService to KRPC

This patch ports the ImpalaInternalService to KRPC.

* ImpalaInternalService is split into two KRPC services. The first,
  ImpalaInternalService, deals with control messages for plan fragment
  instance execution, cancellation and reporting. The second,
  DataStreamService, handles large-payload RPCs for transmitting runtime
  filters and row batches between hosts. The separation allows us to
  dedicate resources to each service, rather than have them compete for
  the same thread pool and queue space.

* In the DataStreamService, all RPCs use 'native'
  protobuf. ImpalaInternalService RPCs remain wrappers of Thrift data
  structures.

* This patch adds support for asynchronous RPCs to the RpcMgr and Rpc
  classes. Previously, Impala used fixed size thread pools + synchronous
  RPCs to achieve some parallelism for 'broadcast' RPCs like filter
  propagation, or a dedicated per-sender+receiver pair thread on the
  sender side in the DataStreamSender case. In this patch, the
  PublishFilter(), CancelPlanFragment() and TransmitData() RPCs are all
  sent asynchronously using KRPC's thread pools.

* As a result, DataStreamSender no longer creates a
  thread-per-connection on the sender side. In this patch, the
  receiver-side still blocks if the receiver is unable to process a new
  row batch. A follow-on patch will change that to send notifications to
  the sender asynchronously, without blocking in the receiver threads.

* A large portion of this patch is the replacement of TRowBatch with its
  Protobuf equivalent, RowBatchPb. The replacement is a literal port of
  the data structure, and row-batch-test, row-batch-list-test and
  row-batch-serialize-benchmark continue to execute without logic
  changes.

* This patch also begins a clean-up of how ImpalaServer instances are created (by
  removing CreateImpalaServer), and clarifying the relationship between
  ExecEnv and ImpalaServer. ImpalaServer now follows the standard
  construct->Init()->Start()->Join() lifecycle that we use for other
  services.

TESTING
-------

* New tests added to rpc-mgr-test.

TO DO
-----

* Re-enable throughput and latency measurements per data-stream sender
  when that information is exposed from KRPC (KUDU-1738).

Change-Id: I95229290566a8ccffd80ed2d74c1c57cf1479238
---
M .clang-format
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/benchmarks/bloom-filter-benchmark.cc
M be/src/benchmarks/expr-benchmark.cc
M be/src/benchmarks/row-batch-serialize-benchmark.cc
M be/src/common/status.cc
M be/src/common/status.h
M be/src/rpc/CMakeLists.txt
M be/src/rpc/common.proto
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc.h
M be/src/rpc/thrift-server-test.cc
D be/src/runtime/backend-client.h
M be/src/runtime/client-cache-types.h
M be/src/runtime/client-cache.cc
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/runtime/data-stream-mgr.cc
M be/src/runtime/data-stream-mgr.h
M be/src/runtime/data-stream-recvr.cc
M be/src/runtime/data-stream-recvr.h
M be/src/runtime/data-stream-sender.cc
M be/src/runtime/data-stream-sender.h
M be/src/runtime/data-stream-test.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/fragment-instance-state.cc
M be/src/runtime/fragment-instance-state.h
M be/src/runtime/plan-fragment-executor.h
M be/src/runtime/row-batch-serialize-test.cc
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/runtime-filter-bank.h
M be/src/runtime/runtime-state.cc
M be/src/runtime/runtime-state.h
M be/src/scheduling/request-pool-service.h
M be/src/scheduling/simple-scheduler-test-util.h
M be/src/service/CMakeLists.txt
M be/src/service/fe-support.cc
M be/src/service/frontend.h
M be/src/service/impala-internal-service.cc
M be/src/service/impala-internal-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
A be/src/service/impala_internal_service.proto
M be/src/service/impalad-main.cc
M be/src/testutil/in-process-servers.cc
M be/src/testutil/in-process-servers.h
M be/src/util/bloom-filter-test.cc
M be/src/util/bloom-filter.cc
M be/src/util/bloom-filter.h
M be/src/util/hdfs-util-test.cc
M common/thrift/CMakeLists.txt
M common/thrift/ImpalaInternalService.thrift
M common/thrift/Results.thrift
59 files changed, 1,400 insertions(+), 1,264 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/5888/1
-- 
To view, visit http://gerrit.cloudera.org:8080/5888
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I95229290566a8ccffd80ed2d74c1c57cf1479238
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson <henry@cloudera.com>

Mime
View raw message