hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Margus Roo <mar...@roo.ee>
Subject Re: Use CodedInputStream.setSizeLimit() to increase the size limit on Read more than 64M content
Date Tue, 09 Dec 2014 14:28:55 GMT
One more thing. I can put larger files than 64M into hbase using java 
API and there is not problem with message size.

Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 480

On 09/12/14 15:53, Margus Roo wrote:
> Hi more additional information
>
> Connecting to bigdata3/192.168.80.232:60020
> IPC Client (155412783) connection to bigdata3/192.168.80.232:60020 
> from margusja: wrote request header call_id: 16 method_name: "Get" 
> request_param: true
> IPC Client (155412783) connection to bigdata3/192.168.80.232:60020 
> from margusja: starting, connections 2
> IPC Client (155412783) connection to bigdata3/192.168.80.232:60020 
> from margusja:*got response header call_id: 16, totalSize: 68266970 
> bytes*
> IPC Client (155412783) connection to bigdata3/192.168.80.232:60020 
> from margusja: closing ipc connection to 
> bigdata3.webmedia.int/192.168.80.232:60020: Protocol message was too 
> large.  May be malicious.  Use CodedInputStream.setSizeLimit() to 
> increase the size limit.
> com.google.protobuf.InvalidProtocolBufferException: Protocol message 
> was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() 
> to increase the size limit.
>     at 
> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
>     at 
> com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
>     at 
> com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
>     at 
> com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$Result.<init>(ClientProtos.java:4086)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$Result.<init>(ClientProtos.java:4050)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$Result$1.parsePartialFrom(ClientProtos.java:4149)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$Result$1.parsePartialFrom(ClientProtos.java:4144)
>     at 
> com.google.protobuf.CodedInputStream.readMessage(CodedInputStream.java:309)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetResponse.<init>(ClientProtos.java:5951)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetResponse.<init>(ClientProtos.java:5898)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetResponse$1.parsePartialFrom(ClientProtos.java:5989)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetResponse$1.parsePartialFrom(ClientProtos.java:5984)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetResponse$Builder.mergeFrom(ClientProtos.java:6282)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$GetResponse$Builder.mergeFrom(ClientProtos.java:6171)
>     at 
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:220)
>     at 
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:912)
>     at 
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:267)
>     at 
> com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:290)
>     at 
> com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:926)
>     at 
> com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:296)
>     at 
> com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:918)
>     at 
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1099)
>     at 
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:726)
> IPC Client (155412783) connection to bigdata3/192.168.80.232:60020 
> from margusja: closed
> IPC Client (155412783) connection to bigdata3/192.168.80.232:60020 
> from margusja: stopped, connections 1
> IPC Client (155412783) connection to bigdata2/192.168.80.70:60020 from 
> margusja: closed
> IPC Client (155412783) connection to bigdata2/192.168.80.70:60020 from 
> margusja: stopped, connections 0
>
> Any hints?
>
> Margus (margusja) Roo
> http://margus.roo.ee
> skype: margusja
> +372 51 480
>
> On 09/12/14 08:45, Margus Roo wrote:
>> Hi
>>
>> I am getting:
>> org.apache.hadoop.hbase.client.RpcRetryingCaller@4d6a54b0, 
>> java.io.IOException: Call to regionserver1/192.168.81.166:60020 
>> failed on local exception: 
>> com.google.protobuf.InvalidProtocolBufferException: Protocol message 
>> was too large.  May be malicious.  Use 
>> CodedInputStream.setSizeLimit() to increase the size limit.
>>
>> I am using Java API
>> ...
>>         // Get image from table
>>         HTablePool pool = new HTablePool(getConf(), 1);
>>         HTableInterface usersTable = pool.getTable(table);
>>
>>         Get g = new Get(Bytes.toBytes(uid));
>>         Result r = usersTable.get(g);
>>
>>         //g.addFamily(Bytes.toBytes("info"));
>>         byte[] b = 
>> r.getValue(Bytes.toBytes(cf),Bytes.toBytes("content"));
>>         String image = Bytes.toString(b);
>>
>>         FileOutputStream imageOutFile = new 
>> FileOutputStream(outFileName);
>>         imageOutFile.write(Base64.decodeBase64(image));
>> ...
>>
>> As I understand then com.google.protobuf is limiting more than 64M 
>> messages.
>> Is that related? https://issues.apache.org/jira/browse/HBASE-11747
>>
>> Is there workaround?
>>
>
>


Mime
View raw message