impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Ho (Code Review)" <>
Subject [Impala-ASF-CR] KUDU-2065, KUDU-2011: Release sidecars on cancellation or timeout
Date Sun, 20 Aug 2017 01:12:03 GMT
Hello Todd Lipcon, Kudu Jenkins,

I'd like you to do a code review.  Please visit

to review the following change.

Change subject: KUDU-2065, KUDU-2011: Release sidecars on cancellation or timeout

KUDU-2065, KUDU-2011: Release sidecars on cancellation or timeout

Previously, when an outbound call times out after transmission
of the request has begun but not finished, the RPC layer will
continue to send the entire payload (including any sidecars) to
the remote destination until the promised number of bytes specified
in the header are sent. This is problematic for the users of the
RPC layer as there is no well defined point in which the sidecars
are no longer referenced by the RPC layer. The only model in which
this would work is for the caller to either transfer or share ownership
of the sidecars with the RPC layer. This has caused some life cycle
issues with row batches in Impala before (e.g. IMPALA-5093).

This change fixes the problem above by modifying the RPC protocol
to allow a mid-transmission RPC call to be cancelled. Specifically,
a new footer is added to all outbound call requests. It contains
a flag, when true, indicates the outbound call was cancelled after
it has started but before it finished. The server will inspect this
flag and ignore inbound calls with this flag set. This footer enables
the caller to relinquish references to the sidecars early when an
outbound call is cancelled or timed-out. Once the call is cancelled
or timed-out, the RPC layer will send the remainder of the bytes for
the request with some dummy bytes. Since the server always ignores
cancelled call requests, it's okay to send random bytes.

To avoid breaking compatibility, a new RPC feature flag REQUEST_FOOTERS
is introduced in this change. During connection negotiation, the
client will always include this flag in its feature set. A server which
supports parsing footer will include REQUEST_FOOTERS in its feature set
if it sees the client also supports it. A client will only send the
footer if the remote server has the RPC feature flag REQUEST_FOOTERS.

Change-Id: I5c8e294090279649082eebc4f6cfb6fe858bbbfc
Reviewed-by: Todd Lipcon <>
Tested-by: Kudu Jenkins
M be/src/kudu/rpc/
M be/src/kudu/rpc/
M be/src/kudu/rpc/connection.h
M be/src/kudu/rpc/
M be/src/kudu/rpc/
M be/src/kudu/rpc/inbound_call.h
M be/src/kudu/rpc/
M be/src/kudu/rpc/outbound_call.h
M be/src/kudu/rpc/
M be/src/kudu/rpc/rpc-test-base.h
M be/src/kudu/rpc/
M be/src/kudu/rpc/rpc_header.proto
M be/src/kudu/rpc/rtest.proto
M be/src/kudu/rpc/
M be/src/kudu/rpc/serialization.h
M be/src/kudu/rpc/
M be/src/kudu/rpc/
M be/src/kudu/rpc/transfer.h
18 files changed, 502 insertions(+), 150 deletions(-)

  git pull ssh:// refs/changes/47/7747/1
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newchange
Gerrit-Change-Id: I5c8e294090279649082eebc4f6cfb6fe858bbbfc
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Michael Ho <>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <>

View raw message