impala-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tarmstr...@apache.org
Subject [1/6] impala git commit: KUDU-2296: Fix deserialization of messages larger than 64MB
Date Wed, 14 Feb 2018 23:21:13 GMT
Repository: impala
Updated Branches:
  refs/heads/master f9971f81a -> b0d3433e3


KUDU-2296: Fix deserialization of messages larger than 64MB

Protobuf's CodedInputStream has a 64MB total byte limit by
default. When trying to deserialize messages larger than
this, ParseMessage() hits this limit and mistakenly
think that the packet is too short. This issue is dormant
due to Kudu's default rpc_max_message_size of 50MB.
However, Impala will be using a larger value for
rpc_max_message_size and requires this fix.

The fix is to override the default 64MB limit by calling
CodedInputStream::SetTotalByteLimit() with the buffer's
size.

Change-Id: I57d3f3ca6ec0aa8be0e67e6a13c4b560c9d2c63a
Reviewed-on: http://gerrit.cloudera.org:8080/9312
Tested-by: Kudu Jenkins
Reviewed-by: Todd Lipcon <todd@apache.org>
Reviewed-on: http://gerrit.cloudera.org:8080/9313
Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
Tested-by: Impala Public Jenkins


Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/d7f2ce10
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/d7f2ce10
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/d7f2ce10

Branch: refs/heads/master
Commit: d7f2ce10b4b77258640be3dfba3bee6cb5031555
Parents: f9971f8
Author: Joe McDonnell <joemcdonnell@cloudera.com>
Authored: Tue Feb 13 14:43:19 2018 -0800
Committer: Impala Public Jenkins <impala-public-jenkins@gerrit.cloudera.org>
Committed: Wed Feb 14 03:01:59 2018 +0000

----------------------------------------------------------------------
 be/src/kudu/rpc/serialization.cc | 4 ++++
 1 file changed, 4 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/impala/blob/d7f2ce10/be/src/kudu/rpc/serialization.cc
----------------------------------------------------------------------
diff --git a/be/src/kudu/rpc/serialization.cc b/be/src/kudu/rpc/serialization.cc
index dbb0fc5..fbc05bc 100644
--- a/be/src/kudu/rpc/serialization.cc
+++ b/be/src/kudu/rpc/serialization.cc
@@ -114,6 +114,10 @@ Status ParseMessage(const Slice& buf,
     << "Got mis-sized buffer: " << KUDU_REDACT(buf.ToDebugString());
 
   CodedInputStream in(buf.data(), buf.size());
+  // Protobuf enforces a 64MB total bytes limit on CodedInputStream by default.
+  // Override this default with the actual size of the buffer to allow messages
+  // larger than 64MB.
+  in.SetTotalBytesLimit(buf.size(), -1);
   in.Skip(kMsgLengthPrefixLength);
 
   uint32_t header_len;


Mime
View raw message