incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Rodriguez <okramma...@gmail.com>
Subject Re: Frame size exceptions occurring with ColumnFamilyInputFormat for very large rows
Date Wed, 28 Nov 2012 04:52:43 GMT
Edward,

I tried:

  cassandra.input.split.size=512

and...<drumroll/>. It worked. Splendid.

For others that might find this problem via Google. The default in
ConfigHelper is:

    private static final String INPUT_SPLIT_SIZE_CONFIG =
"cassandra.input.split.size";
    private static final int DEFAULT_SPLIT_SIZE = 64 * 1024;

Thus -- 64 megs. For my situation, I had to reduce it to 1/2 a meg (at 1
meg, I still had a frame size exception).

Anywho, thank you Edward,
Marko.

http://thinkaurelius.com

On Tue, Nov 27, 2012 at 9:27 PM, Edward Capriolo <edlinuxguru@gmail.com>wrote:

> Have you tried lowering the slice size?
>
>
> On Tue, Nov 27, 2012 at 11:12 PM, Marko Rodriguez <okrammarko@gmail.com>wrote:
>
>> Hi,
>>
>> Even when setting it to 32m in cassandra.yaml (and restarting Cassandra),
>> the same problem emerges -- its as if Cassandra doesn't register the update
>> (its always locked at 16mb). And I know that Cassandra is reading the
>> property from cassandra.yaml because if I do -1, it complains saying it
>> must be a positive value.
>>
>> Apologies for the back and forth --- though I have no obvious way forward
>> for myself.
>>
>> Thank you,
>> Marko.
>>
>> http://markorodriguez.com
>>
>> P.S. Is a brontobyte an order of magnitude less than a tyranobyte?
>>
>> On Nov 27, 2012, at 8:26 PM, Edward Capriolo wrote:
>>
>> Thrift has to buffer the packet into memory so setting it to 1,000,000
>> brontobytes is a bad idea.
>>
>> On Tue, Nov 27, 2012 at 9:17 PM, Marko Rodriguez <okrammarko@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> I was wondering if anyone had an answer to my previous message below.
>>>
>>> Seems another is having the same problem, but unfortunately with no
>>> response as well.
>>>
>>> http://mail-archives.apache.org/mod_mbox/cassandra-user/201211.mbox/%3C509A4A1F.8070506@semantico.com%3E
>>>  Any help would be much appreciated.
>>>
>>> Thank you,
>>> Marko.
>>>
>>> http://markorodriguez.com
>>>
>>> On Nov 9, 2012, at 3:02 PM, Marko Rodriguez wrote:
>>>
>>> Hello,
>>>
>>> I am trying to run a Hadoop job that pulls data out of Cassandra via
>>> ColumnFamilyInputFormat. I am getting a "frame size" exception. To remedy
>>> that, I have set both the thrift_framed_transport_size_in_mb and
>>> thrift_max_message_length_in_mb to an "infinite" amount at 100000mb on all
>>> nodes. Moreover, I have restarted the cluster and the cassandra.yaml files
>>> have been reloaded.
>>>
>>> However, I am still getting:
>>>
>>> 12/11/09 21:39:52 INFO mapred.JobClient:  map 62% reduce 0%
>>> 12/11/09 21:40:09 INFO mapred.JobClient: Task Id :
>>> attempt_201211082011_0015_m_000479_2, Status : FAILED
>>> java.lang.RuntimeException:
>>> org.apache.thrift.transport.TTransportException: Frame size (30046945)
>>> larger than max length (16384000)!
>>>  at
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400)
>>> at
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
>>>  at
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:324)
>>> at
>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>>>  at
>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>>> at
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:189)
>>>
>>> Question: Why is 16384000 bytes (I assume) !=  100000mb?
>>>
>>> Next, I made this parameter true as a last hail mary attempt:
>>> cassandra.input.widerows=true
>>> ...still with no luck.
>>>
>>> Does someone know what I might be missing?
>>>
>>> Thank you very much for your time,
>>> Marko.
>>>
>>> http://markorodriguez.com
>>>
>>>
>>>
>>
>>
>


-- 
http://markorodriguez.com

Mime
View raw message