impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sailesh Mukil (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4875: Bound maximum statestore topic update size to 2GB.
Date Fri, 14 Jul 2017 22:10:57 GMT
Sailesh Mukil has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/7425

Change subject: IMPALA-4875: Bound maximum statestore topic update size to 2GB.
......................................................................

IMPALA-4875: Bound maximum statestore topic update size to 2GB.

This patch enforces a maximum topic update size for the statestore to be
2GB. If that limit is exceeded, the statestore will split topic updates
over several RPCs to the subscriber. If a single topic key + value
exceeds the limit, it will be dropped by the statestore when the
subscriber tries to publish it.

Doing so introduces the possibility that a catalog update (all updates
pertaining to a single catalog version) might be split across two
separate UpdateState() RPCs. To ensure that all updates are flushed to
the catalog FE at the same time, we buffer updates until a new object of
type CATALOG is received.

Remove the special treatment of deletions in the UpdateState()
RPC. Previously they were not sent with their version.

New statestore tests added to ensure:

* Update sizes don't exceed the configured max.
* 'from' version is properly handled (not previously tested) so that
  subscribers can process only a prefix of the update log.

TODO : Check min subscriber version

Change-Id: Icbe42704d3c98d9619e3737786b75ee67fb2a264
---
M be/src/catalog/catalog-server.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/scheduler-test-util.cc
M be/src/scheduling/scheduler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/statestore/statestore-test.cc
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M common/thrift/StatestoreService.thrift
11 files changed, 308 insertions(+), 163 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/7425/1
-- 
To view, visit http://gerrit.cloudera.org:8080/7425
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Icbe42704d3c98d9619e3737786b75ee67fb2a264
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Sailesh Mukil <sailesh@cloudera.com>
Gerrit-Reviewer: Henry Robinson <henry@cloudera.com>
Gerrit-Reviewer: Michael Ho <kwho@cloudera.com>

Mime
View raw message