kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhou Kang <zhoukan...@outlook.com>
Subject KYLIN-3973: InvalidProtocolBufferException
Date Mon, 09 Sep 2019 07:28:37 GMT
Hi, guys

I reproduce this problem: https://issues.apache.org/jira/browse/KYLIN-3973

My env is Kylin 2.5.2 + HBase 2.2. After reading the code of hbase, I found the problem but
don’t know the best method to resolve it.

in hbase coprocessor rpc, data pass to hbase client by CodedInputStream.newInstance(). in
hbase 0.98 the input is byte[], and hbase 2.2 is InputStream. When protobuf read the input
data, byte[] will not trigger the size check in function refillBuffer. But in hbase 2.2, the
inputStream will refill many times and finnaly hit the limit of sizeLimit(DEFAULT_SIZE_LIMIT).
So the solution is enlarge this value.

Extra, DEFAULT_SIZE_LIMIT is 64M in protobuf 2.5.0, which is changed to Integer.MAX_VALUE
in protobuf3.


A simple way to fix bug is useing input.setSizeLimit() to modify this limit. But coprocessor/endpoint/generated/CubeVisitProtos.java
is an auto generated file.

  Another way is changing com.google<http://com.google>.protobuf:2.5.0 to a shaded package.
Some other lib(such as hbase) choose this way.


I need your suggestion ~


Thank you.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message