incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Rodriguez <okramma...@gmail.com>
Subject Re: Frame size exceptions occurring with ColumnFamilyInputFormat for very large rows
Date Thu, 29 Nov 2012 01:03:30 GMT
Hi,

Yes. In cassandra.yaml, I set the thrift_max_message_length_in_mb parameter
to 32mb when I went with Edward's recommendation to reduce the split size.
I don't understand the relationship between these two parameters... :/ But
just trial and error for the time I had -- 32mb / 512kb was what worked for
my wide column Hadoop job.

Marko.


On Wed, Nov 28, 2012 at 5:04 AM, Jason Wee <peichieh@gmail.com> wrote:

> do you mean the setting 'thrift_max_message_length_in_mb' in
> cassandra.yaml that you changed from 16mb to 32mb?
>
> /jason
>
>
> On Wed, Nov 28, 2012 at 12:12 PM, Marko Rodriguez <okrammarko@gmail.com>wrote:
>
>> Hi,
>>
>> Even when setting it to 32m in cassandra.yaml (and restarting Cassandra),
>> the same problem emerges -- its as if Cassandra doesn't register the update
>> (its always locked at 16mb). And I know that Cassandra is reading the
>> property from cassandra.yaml because if I do -1, it complains saying it
>> must be a positive value.
>>
>> Apologies for the back and forth --- though I have no obvious way forward
>> for myself.
>>
>> Thank you,
>> Marko.
>>
>> http://markorodriguez.com
>>
>> P.S. Is a brontobyte an order of magnitude less than a tyranobyte?
>>
>> On Nov 27, 2012, at 8:26 PM, Edward Capriolo wrote:
>>
>> Thrift has to buffer the packet into memory so setting it to 1,000,000
>> brontobytes is a bad idea.
>>
>> On Tue, Nov 27, 2012 at 9:17 PM, Marko Rodriguez <okrammarko@gmail.com>wrote:
>>
>>> Hello,
>>>
>>> I was wondering if anyone had an answer to my previous message below.
>>>
>>> Seems another is having the same problem, but unfortunately with no
>>> response as well.
>>>
>>> http://mail-archives.apache.org/mod_mbox/cassandra-user/201211.mbox/%3C509A4A1F.8070506@semantico.com%3E
>>>  Any help would be much appreciated.
>>>
>>> Thank you,
>>> Marko.
>>>
>>> http://markorodriguez.com
>>>
>>> On Nov 9, 2012, at 3:02 PM, Marko Rodriguez wrote:
>>>
>>> Hello,
>>>
>>> I am trying to run a Hadoop job that pulls data out of Cassandra via
>>> ColumnFamilyInputFormat. I am getting a "frame size" exception. To remedy
>>> that, I have set both the thrift_framed_transport_size_in_mb and
>>> thrift_max_message_length_in_mb to an "infinite" amount at 100000mb on all
>>> nodes. Moreover, I have restarted the cluster and the cassandra.yaml files
>>> have been reloaded.
>>>
>>> However, I am still getting:
>>>
>>> 12/11/09 21:39:52 INFO mapred.JobClient:  map 62% reduce 0%
>>> 12/11/09 21:40:09 INFO mapred.JobClient: Task Id :
>>> attempt_201211082011_0015_m_000479_2, Status : FAILED
>>> java.lang.RuntimeException:
>>> org.apache.thrift.transport.TTransportException: Frame size (30046945)
>>> larger than max length (16384000)!
>>>  at
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.maybeInit(ColumnFamilyRecordReader.java:400)
>>> at
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:406)
>>>  at
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader$StaticRowIterator.computeNext(ColumnFamilyRecordReader.java:324)
>>> at
>>> com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>>>  at
>>> com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>>> at
>>> org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(ColumnFamilyRecordReader.java:189)
>>>
>>> Question: Why is 16384000 bytes (I assume) !=  100000mb?
>>>
>>> Next, I made this parameter true as a last hail mary attempt:
>>> cassandra.input.widerows=true
>>> ...still with no luck.
>>>
>>> Does someone know what I might be missing?
>>>
>>> Thank you very much for your time,
>>> Marko.
>>>
>>> http://markorodriguez.com
>>>
>>>
>>>
>>
>>
>


-- 
http://markorodriguez.com

Mime
View raw message