crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CRUNCH-534) Protobuf Size Limit Exception
Date Thu, 02 Jul 2015 02:23:04 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Micah Whitacre updated CRUNCH-534:
----------------------------------
    Description: 
With HBase's switch to ProtocolBuffers there is now an imposed restriction on the size of
data (64MB) which can be stored in protocol buffers and therefore Cells/Puts/etc.  

{quote}
Call to  failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol
message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase
the size limit.
{quote}

While a single cell being that big seems extreme this limit also counts towards Result objects
so an entire row.

It is definitely an HBase problem[1] but Crunch also might have changes needed since we use
protobuf serialization in our HBaseTypes.[2]  So might need to make sure error isn't only
on our side as they work through the issue or might need to look at MOB support[3].

[1] - https://issues.apache.org/jira/browse/HBASE-13825
[2] - https://github.com/apache/crunch/blob/d176778cf803374506cb7743069a05e28e07e2cf/crunch-hbase/src/main/java/org/apache/crunch/io/hbase/HBaseTypes.java#L33
[3] - https://issues.apache.org/jira/browse/HBASE-11339

  was:
With HBase's switch to ProtocolBuffers there is now an imposed restriction on the size of
data which can be stored in protocol buffers and therefore Cells/Puts/etc.  

{quote}
Call to  failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Protocol
message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase
the size limit.
{quote}

It is definitely an HBase problem[1] but Crunch also might have changes needed since we use
protobuf serialization in our HBaseTypes.[2]  So might need to make sure error isn't only
on our side as they work through the issue or might need to look at MOB support[3].

[1] - https://issues.apache.org/jira/browse/HBASE-13825
[2] - https://github.com/apache/crunch/blob/d176778cf803374506cb7743069a05e28e07e2cf/crunch-hbase/src/main/java/org/apache/crunch/io/hbase/HBaseTypes.java#L33
[3] - https://issues.apache.org/jira/browse/HBASE-11339


> Protobuf Size Limit Exception 
> ------------------------------
>
>                 Key: CRUNCH-534
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-534
>             Project: Crunch
>          Issue Type: Bug
>          Components: IO
>            Reporter: Micah Whitacre
>
> With HBase's switch to ProtocolBuffers there is now an imposed restriction on the size
of data (64MB) which can be stored in protocol buffers and therefore Cells/Puts/etc.  
> {quote}
> Call to  failed on local exception: com.google.protobuf.InvalidProtocolBufferException:
Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to
increase the size limit.
> {quote}
> While a single cell being that big seems extreme this limit also counts towards Result
objects so an entire row.
> It is definitely an HBase problem[1] but Crunch also might have changes needed since
we use protobuf serialization in our HBaseTypes.[2]  So might need to make sure error isn't
only on our side as they work through the issue or might need to look at MOB support[3].
> [1] - https://issues.apache.org/jira/browse/HBASE-13825
> [2] - https://github.com/apache/crunch/blob/d176778cf803374506cb7743069a05e28e07e2cf/crunch-hbase/src/main/java/org/apache/crunch/io/hbase/HBaseTypes.java#L33
> [3] - https://issues.apache.org/jira/browse/HBASE-11339



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message