crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Esteban Gutierrez (JIRA)" <>
Subject [jira] [Commented] (CRUNCH-534) Protobuf Size Limit Exception
Date Tue, 14 Jul 2015 18:42:05 GMT


Esteban Gutierrez commented on CRUNCH-534:

Crunch is only importing those dependencies and it should be agnostic about how we handle
that internally by HBase. I have created  HBASE-14076  to fix this on HBase.

> Protobuf Size Limit Exception 
> ------------------------------
>                 Key: CRUNCH-534
>                 URL:
>             Project: Crunch
>          Issue Type: Bug
>          Components: IO
>            Reporter: Micah Whitacre
> With HBase's switch to ProtocolBuffers there is now an imposed restriction on the size
of data (64MB) which can be stored in protocol buffers and therefore Cells/Puts/etc.  
> {quote}
> Call to  failed on local exception:
Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to
increase the size limit.
> {quote}
> While a single cell being that big seems extreme this limit also counts towards Result
objects so an entire row.
> It is definitely an HBase problem[1] but Crunch also might have changes needed since
we use protobuf serialization in our HBaseTypes.[2]  So might need to make sure error isn't
only on our side as they work through the issue or might need to look at MOB support[3].
> [1] -
> [2] -
> [3] -

This message was sent by Atlassian JIRA

View raw message