From commits-return-3165-archive-asf-public=cust-asf.ponee.io@impala.apache.org Thu Feb 15 00:21:14 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 67B74180621 for ; Thu, 15 Feb 2018 00:21:14 +0100 (CET) Received: (qmail 33684 invoked by uid 500); 14 Feb 2018 23:21:13 -0000 Mailing-List: contact commits-help@impala.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@impala.apache.org Delivered-To: mailing list commits@impala.apache.org Received: (qmail 33665 invoked by uid 99); 14 Feb 2018 23:21:13 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Feb 2018 23:21:13 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 6BC2EDFC32; Wed, 14 Feb 2018 23:21:13 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: tarmstrong@apache.org To: commits@impala.apache.org Date: Wed, 14 Feb 2018 23:21:13 -0000 Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: [1/6] impala git commit: KUDU-2296: Fix deserialization of messages larger than 64MB Repository: impala Updated Branches: refs/heads/master f9971f81a -> b0d3433e3 KUDU-2296: Fix deserialization of messages larger than 64MB Protobuf's CodedInputStream has a 64MB total byte limit by default. When trying to deserialize messages larger than this, ParseMessage() hits this limit and mistakenly think that the packet is too short. This issue is dormant due to Kudu's default rpc_max_message_size of 50MB. However, Impala will be using a larger value for rpc_max_message_size and requires this fix. The fix is to override the default 64MB limit by calling CodedInputStream::SetTotalByteLimit() with the buffer's size. Change-Id: I57d3f3ca6ec0aa8be0e67e6a13c4b560c9d2c63a Reviewed-on: http://gerrit.cloudera.org:8080/9312 Tested-by: Kudu Jenkins Reviewed-by: Todd Lipcon Reviewed-on: http://gerrit.cloudera.org:8080/9313 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/impala/repo Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/d7f2ce10 Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/d7f2ce10 Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/d7f2ce10 Branch: refs/heads/master Commit: d7f2ce10b4b77258640be3dfba3bee6cb5031555 Parents: f9971f8 Author: Joe McDonnell Authored: Tue Feb 13 14:43:19 2018 -0800 Committer: Impala Public Jenkins Committed: Wed Feb 14 03:01:59 2018 +0000 ---------------------------------------------------------------------- be/src/kudu/rpc/serialization.cc | 4 ++++ 1 file changed, 4 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/impala/blob/d7f2ce10/be/src/kudu/rpc/serialization.cc ---------------------------------------------------------------------- diff --git a/be/src/kudu/rpc/serialization.cc b/be/src/kudu/rpc/serialization.cc index dbb0fc5..fbc05bc 100644 --- a/be/src/kudu/rpc/serialization.cc +++ b/be/src/kudu/rpc/serialization.cc @@ -114,6 +114,10 @@ Status ParseMessage(const Slice& buf, << "Got mis-sized buffer: " << KUDU_REDACT(buf.ToDebugString()); CodedInputStream in(buf.data(), buf.size()); + // Protobuf enforces a 64MB total bytes limit on CodedInputStream by default. + // Override this default with the actual size of the buffer to allow messages + // larger than 64MB. + in.SetTotalBytesLimit(buf.size(), -1); in.Skip(kMsgLengthPrefixLength); uint32_t header_len;